Fakultät für informatik informatik 12 technische universität dortmund Evaluation and Validation Peter Marwedel TU Dortmund, Informatik 12 Germany 2009/12/09.

Präsentation zum Thema: "Fakultät für informatik informatik 12 technische universität dortmund Evaluation and Validation Peter Marwedel TU Dortmund, Informatik 12 Germany 2009/12/09."—  Präsentation transkript:

1 fakultät für informatik informatik 12 technische universität dortmund Evaluation and Validation Peter Marwedel TU Dortmund, Informatik 12 Germany 2009/12/09 Graphics: © Alexandra Nolte, Gesine Marwedel, 2003 © These slides use Microsoft cliparts. All Microsoft restrictions apply.

2 - 2 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Structure of this course 2: Specification 3: ES-hardware 4: system software (RTOS, middleware, …) 8: Test 5: Validation & Evaluation (energy, cost, performance, …) 7: Optimization 6: Application mapping Application Knowledge Design repository Design Numbers denote sequence of chapters

3 - 3 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Performance evaluation  Estimated cost and performance values: Difficult to generate sufficiently precise estimates; Balance between run-time and precision  Accurate cost and performance values: Can be done with normal tools (such as compilers). As precise as the input data is. π x We need to compute average and worst case execution times

4 - 4 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Thiele’s real-time calculus (RTC)/ Modular performance analysis (MPA) - Arrival curves - 1 2 3  T 2T3T 1 2 3  T 2T3T T-JT+J periodic event streamperiodic event stream with jitter Arrival curves describe the maximum and minimum number of events arriving in some time interval  Examples

5 - 5 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Thiele’s real-time calculus (RTC)/ Modular performance analysis (MPA) - Service curves - Service curves  u resp.  ℓ describe the maximum and minimum service capacity available in some time interval  s T bandwidth b TDMA bus TsT-sT+s bsbs 2T Example:

6 - 6 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Thiele’s real-time calculus (RTC)/ Modular performance analysis (MPA) - Workload characterization -  u resp.  ℓ describe the maximum and minimum service capacity required as a function of the number e of events Example 1 2 3 4 8 12 16 e WCET=4 BCET=3  u u  ℓ ℓ (Defined only for an integer number of events)

7 - 7 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Workload required for incoming stream Incoming workload Upper and lower bounds on the number of events

8 - 8 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Thiele’s real-time calculus (RTC)/ Modular performance analysis (MPA) - System of real time components - RTC RTC’ RTC” … Incoming event streams and available capacity are transformed by real-time components: Theoretical results allow the computation of properties of outgoing streams 

9 - 9 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Thiele’s real-time calculus (RTC)/ Modular performance analysis (MPA) - Transformation of arrival and service curves Resulting arrival curves: Remaining service curves: Where:

10 - 10 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Thiele’s real-time calculus Remarks  Details of the proofs can be found in relevant references  Results also include bounds on buffer sizes and on maximum latency.  Theory has been extended into various directions, e.g. for computing remaining battery capacities

11 - 11 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Application: In-Car Navigation System Car radio with navigation system User interface needs to be responsive Traffic messages (TMC) must be processed in a timely way Several applications may execute concurrently © Thiele, ETHZ

12 - 12 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 System Overview NAVRAD MMI DB Communication © Thiele, ETHZ

13 - 13 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 NAVRAD MMI DB Communication Use case 1: Change Audio Volume < 200 ms < 50 ms © Thiele, ETHZ

14 - 14 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Use case 1: Change Audio Volume © Thiele, ETHZ Communication Resource Demand

15 - 15 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 NAVRAD MMI DB Communication < 200 ms Use case 2: Lookup Destination Address © Thiele, ETHZ

16 - 16 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Use case 2: Lookup Destination Address © Thiele, ETHZ

17 - 17 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 NAVRAD MMI DB Communication Use case 3: Receive TMC Messages < 1000 ms © Thiele, ETHZ

18 - 18 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Use case 3: Receive TMC Messages © Thiele, ETHZ

19 - 19 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Proposed Architecture Alternatives NAV RAD MMI 22 MIPS 11 MIPS 113 MIPS NAVRAD MMI 22 MIPS 11 MIPS113 MIPS RAD 260 MIPS NAV MMI 22 MIPS RAD 130 MIPS MMI NAV 113 MIPS MMI 260 MIPS RAD NAV 72 kbps 57 kbps 72 kbps (A) (E)(D)(C) (B) © Thiele, ETHZ

20 - 20 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Step 1: Environment (Event Steams) Event Stream Model e.g. Address Lookup (1 event / sec) uu ll [s] [events] 1 1 © Thiele, ETHZ

21 - 21 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Step 2: Architectural Elements Event Stream Model e.g. Address Lookup (1 event / sec) Resource Model e.g. unloaded RISC CPU (113 MIPS) ℓ=uℓ=u uu ℓℓ [s] [MIPS] [events] 1 1 1 113 © Thiele, ETHZ

22 - 22 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Step 3: Mapping / Scheduling Rate Monotonic Scheduling (Pre-emptive fixed priority scheduling):  Priority 1:Change Volume (p=1/32 s)  Priority 2:Address Lookup (p=1 s)  Priority 3:Receive TMC (p=6 s) © Thiele, ETHZ

23 - 23 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Step 4: Performance Model CPU1BUSCPU3 CPU2 Change Volume Receive TMC     NAVRAD MMI Address Lookup   MMI NAVRAD © Thiele, ETHZ

24 - 24 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Step 5: Analysis CPU1BUSCPU3 CPU2 Change Volume Receive TMC     NAVRAD MMI Address Lookup   MMI NAVRAD © Thiele, ETHZ

25 - 25 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Analysis – Design Question 1 How do the proposed system architectures compare in respect to end-to-end delays? © Thiele, ETHZ

26 - 26 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 End-to-end delays: (A)(E)(D)(C)(B) Analysis – Design Question 1 © Thiele, ETHZ

27 - 27 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Analysis – Design Question 2 How robust is architecture A? Where is the bottleneck of this architecture? NAVRAD MMI 22 MIPS 11 MIPS113 MIPS 72 kbps (A) © Thiele, ETHZ

28 - 28 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Sensitivity to input rate: Sensitivity to resource capacity: Analysis – Design Question 2 NAV RAD MMI © Thiele, ETHZ

29 - 29 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 TMC delay vs. MMI processor speed: Analysis – Design Question 2 NAVRAD MMI 22 MIPS 11 MIPS113 MIPS 72 kbps (A) 26.4 MIPS © Thiele, ETHZ

30 - 30 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Analysis – Design Question 3 Architecture D is chosen for further investigation. How should the processors be dimensioned? RAD 130 MIPS MMI NAV 113 MIPS 72 kbps (D) © Thiele, ETHZ

31 - 31 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 RAD 130 MIPS MMI NAV 113 MIPS 72 kbps 33 MIPS 29 MIPS Analysis – Design Question 3 d max = 200 d max = 1000 d max = 50 d max = 200 © Thiele, ETHZ

32 - 32 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Conclusions  Easy to construct models (~ half day)  Evaluation speed is fast and linear to model complexity (~ 1s per evaluation)  Needs little information to construct early models (Fits early design cycle very well)  Even though involved mathematics is very complex, the method is easy to use (Language of engineers) © Thiele, ETHZ

33 - 33 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Acknowledgement and References The presentation contains contributions by  Samarjit Chakraborty (NUS)  Simon Künzli, Ernesto Wandeler, Alexander Maxiaguine (ETHZ)  Andreas Herkersdorf, Patricia Sagmeister (IBM)  Jonas Greutert (Netmodule) Many publications are available from http://www.tik.ee.ethz.ch/~thiele © Thiele, ETHZ

34 - 34 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 SYMTA SYMTA (Ernst, TU Braunschweig) works with standard streams (periodic, periodic with jitter streams) and focuses on computing end-to-end guarantees in multiprocessor based systems See www.symtavision.com

35 - 35 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Worst/best case execution times (1) WCET Actually possible worst case Observed execution time Actually best possible ex. time BCET' WCET’ (tighter bound) BCET t Other authors WCET BCET Estimated WCET Estimated BCET Feasible execution times Observed WCET Definition used in this course Next slide

36 - 36 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Worst/best case execution times (2) Requirements on WCET estimates:  Safeness: WCET  WCET EST !  Tightness: WCET EST – WCET  minimal © h. falk/p. marwedel WCET EST

37 - 37 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Worst case execution times (3) Complexity:  in the general case: undecidable if a bound exists.  for restricted programs: simple for “old“ architectures, very complex for new architectures with pipelines, caches, interrupts, virtual memory, etc. Approaches:  for hardware: requires detailed timing behavior  for software: requires availability of machine programs; complex analysis (see, e.g., www.absint.de) Presentation by R. Wilhelm @ FEST (DVD in German), starting at min. 31:35 min.-47:00

38 - 38 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 WCET estimation: AiT (AbsInt) Executable program CFG-Builder Loop Trafo AIP File ILP-Generator ILP-Solver Evaluation WCET Visuali- zation Loop Bounds Static analyzer Value Analyzer Cache/Pipeline Analyzer PER-File CRL-File Path Analysis

39 - 39 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 WCET estimation for caches Behavior at program joins Worst case Best case {c}{e}{a}{d} {a}{}{c,f}{d} {c}{e}{a}{d} {a}{c,f}{}{d} {} {a,c}{d} {a,c}{e,f}{}{d} Intersection+max. age Union+min. age TagIndexOffset LRU-based replacement ==== Address Movie: Presentation by Reinhard Wilhelm @ Freiburg ES days (German)

40 - 40 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Energy models  Tiwari (1994): Energy consumption within processors  Simunic (1999): Using values from data sheets. Allows modeling of all components, but not very precise.  Russell, Jacome (1998): Measurements for 2 fixed configurations  Steinke et al., UniDo (2001): mixed model using measurements and prediction  Jouppi (1996): Energy consumption of caches predicted by CACTI.

41 - 41 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Steinke’s model E.g.: ATMEL board with ARM7TDMI and ext. SRAM Data Memory Instruction Memory IAddr VDD ALU Multi- plier Barrel Shifter Register File Instr. Decoder & Control Logic Instr Imm Reg Value Reg# Opcode ARM7 DAddr mA Data Instr

42 - 42 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 CACTI model Comparison with SPICE Cache model used http://research.compaq.com/wrl/people/jouppi/CACTI.html

43 - 43 - technische universität dortmund fakultät für informatik  p. marwedel, informatik 12, 2009 Summary Performance analysis  Simulation-based techniques Trade-off between speed and accuracy (  end of chapter 2)  Formal performance analysis Thiele’s real-time calculus (RTC)/Modular performance analysis (MPA) -Using bounds on the number of events in input streams -Using bounds on available processing capability -Derives bounds on the number of events in output streams -Derives bound on remaining processing capability, buffer sizes, … -Examples demonstrate design procedure based on RTC  Evaluation of objective “energy/power consumption”