Conclusion Introduction Toolchain Experiments Solution Perspectives

Parallelizing Quickrefe ELS 2019 E

Didier VernaEPITA / LRDE

didier@lrde.epita.fr

lrde/~didier @didierverna didier.verna in/didierverna

Conclusion Introduction Toolchain Experiments Solution Perspectives

Conclusion

I Quickref parallelizedI Performance improvement: 4x

Thank you! Any questions?

Parallelizing Quickref / ELS 2019 – Didier Verna 2/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Introduction

I Reference manuals for CL(Quicklisp) libraries

I quickref.common-lisp.netor local builds

I Originally: sequential loopover 1700+ libraries

I 1h30 – 7hdepending on the conditions

I Parallelism worth investigating

Parallelizing Quickref / ELS 2019 – Didier Verna 3/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Plan

Toolchain

Experiments

Parallel Solution

Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 4/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Plan

Toolchain

Experiments

Parallel Solution

Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 5/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Toolchain

Quicklisp foo/ Declt foo.texi Makeinfo foo.html

Important remarks:I Declt works by introspection

I Compilation / loading (of dependencies) may be requiredI Avoid loading 1700+ libraries in the same Lisp image!I Run Declt in external processes

I Makeinfo is a Perl/C scriptI Ditto

Parallelizing Quickref / ELS 2019 – Didier Verna 6/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Plan

Toolchain

Experiments

Parallel Solution

Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 7/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Scenarios

I Quickref OptionsI Libraries and Update

I full / installed-onlyI download / compilation may occur

I Compilation cache PolicyI global / localI global cache may cause problems

I Scenarios1. All libraries already compiled (1h 27m)2. Nothing compiled, global compilation cache (1h 51m)3. Nothing compiled, local compilation cache (7h 01m)

Parallelizing Quickref / ELS 2019 – Didier Verna 8/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Time Distribution

Makeinfo48 % Declt

48 %

Loading4 %

1

Makeinfo40 %

Declt41 %

Loading3 %

Compilation16 %

2

Makeinfo10 %

Declt10 %

Loading1 %

Compilation79 %

3

Scenario Texinfo Html1 52% 48%2 60% 40%3 90% 10%

Parallelizing Quickref / ELS 2019 – Didier Verna 9/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Plan

Toolchain

Experiments

Parallel Solution

Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 10/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Parallel Solution

Library Pool Declt

Declt

Declt

Texinfo Files Makeinfo

Makeinfo

Makeinfo

HTML Files

I Usable in all scenariosI Dependency graph management cheapI Scenario 2

I Best results: 4 Declt threads / 4 Makeinfo threadsI Total time: 28m 17s (25% of sequential time)

Parallelizing Quickref / ELS 2019 – Didier Verna 11/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6

1 2

4 3

Library Pool Done2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6

1 2

4 3

Library Pool

1 2

Done2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6

1 2

4 3

Library Pool

1 3

Done2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6

1 2

4 3

Library Pool

4

Done2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6

1 2

4 3

Library Pool

5

Done2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6

1 2

4 3

Library Pool

6

Done2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Library Pool Management

Dependency Graph

5 6

1 2

4 3

Library Pool Done2 1 3 4 5 6

Parallelizing Quickref / ELS 2019 – Didier Verna 12/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Plan

Toolchain

Experiments

Parallel Solution

Discussion and Perspectives

Parallelizing Quickref / ELS 2019 – Didier Verna 14/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Discussion and Perspectives

I Alternative AlgorithmsI Dependency Management Issues

I Based on static information provided by QuicklispI Fragile, not always correct

I CPU vs. I/O ConsumptionI 4x is a bit disappointingI Open the Declt and Makeinfo black boxes

I SJF-like SchedulingI Very difficult to figure out where complexity comes fromI Collect timings and use them in next run

I SSD!

Parallelizing Quickref / ELS 2019 – Didier Verna 15/16

Conclusion Introduction Toolchain Experiments Solution Perspectives

Acknowledgments

I Initial code base: Antoine MartinI Author index and parallel algorithm #4: Antoine HacquardI Hosting (code & website): CLF / common-lisp.net

Parallelizing Quickref / ELS 2019 – Didier Verna 16/16