Pipeline

Overview

The rev.ng pipeline is organized into branches. Each branch is a sequence of tasks. A branch starts from the end of another branch (its parent), so the branches form a tree; the place where a branch departs from its parent is a branching point.

Each task is either:

• a pipe, which transforms the contents of one or more containers;
• a savepoint, which caches the contents of some containers so that later runs can resume the pipeline from there.

A task can also expose an artifact (an output the user can request) or run an analysis (which refines the model).

The following graph shows the branches, their branching points, and the artifacts and analyses attached to each of them. It uses this notation:

• a small empty circle is a branch point;
• a rounded box is an artifact;
• a folder is an artifact produced at a savepoint;
• a dashed box is an analysis.

Containers

The pipeline declares the following containers:

• binaries-container (type: BinariesContainer)
• llvm-root (type: LLVMRootContainer)
• cfg-map (type: CFGMap)
• llvm-functions (type: LLVMFunctionContainer)
• hexdump (type: HexDumpContainer)
• assembly-internal (type: AssemblyInternalContainer)
• assembly (type: AssemblyContainer)
• object-file (type: ObjectFileContainer)
• translated (type: TranslatedContainer)
• llvm-root-with-functions (type: LLVMRootContainer)
• clift-functions (type: CliftFunctionContainer)
• decompile-c (type: PTMLCFunctionContainer)
• decompiled-single-file (type: PTMLCContainer)
• tagless-decompile-c (type: PTMLCFunctionContainer)
• tagless-decompiled-single-file (type: PTMLCContainer)
• recompilable-archive (type: RecompilableArchiveContainer)
• llvm-root-merged (type: LLVMRootContainer)
• cross-relations (type: CrossRelationsContainer)
• call-graph (type: CallGraphContainer)
• call-graph-slice (type: CallGraphSliceContainer)
• function-cfg (type: FunctionControlFlowContainer)
• clift-types-and-globals (type: CliftModuleContainer)
• type-and-global-header (type: PTMLCContainer)
• helper-header (type: PTMLCContainer)
• single-type (type: CTypeContainer)
• tagless-type-and-global-header (type: PTMLCContainer)
• tagless-helper-header (type: PTMLCContainer)

Branches

lift branch

Tasks:

• import-files(binaries-container)
  • Analysis: parse-binary
• lift(binaries-container, llvm-root)
• emit-segment-references(llvm-root)
• pure-llvm-passes-root-pipe(llvm-root)
• savepoint("lifted", llvm-root)
  • Artifact: lift
  • Analysis: detect-abi

collect-cfg branch

Starts from: lift

Tasks:

• collect-cfg(llvm-root, cfg-map)
• savepoint("cfg-computed", cfg-map)
  • Artifact: emit-cfg

isolate branch

Starts from: collect-cfg

Tasks:

• isolate(cfg-map, llvm-root, llvm-functions)
• attach-debug-info(cfg-map, llvm-functions)
• inline-always-inline-functions(llvm-functions)
• savepoint("isolate", llvm-functions)
  • Artifact: isolate

segregate-stack-accesses branch

Starts from: isolate

Tasks:

• enforce-abi(cfg-map, llvm-functions)
• pure-llvm-passes-pipe(llvm-functions)
• promote-csvs(llvm-functions)
• pure-llvm-passes-pipe(llvm-functions)
• attach-debug-info(cfg-map, llvm-functions)
• promote-csvs(llvm-functions)
• pure-llvm-passes-pipe(llvm-functions)
• savepoint("enforce-abi", llvm-functions)
  • Artifact: enforce-abi
• pure-llvm-passes-pipe(llvm-functions)
• remove-lifting-artifacts(llvm-functions)
• promote-init-csv-to-undef(llvm-functions)
• inject-stack-size-probes-at-callsites(llvm-functions)
• promote-stack-pointer(llvm-functions)
• pure-llvm-passes-pipe(llvm-functions)
• simplify-switch(binaries-container, llvm-functions)
• savepoint("simplify-switch", llvm-functions)
  • Artifact: simplify-switch
• pure-llvm-passes-pipe(llvm-functions)
• savepoint("detect-stack-size", llvm-functions)
  • Analysis: detect-stack-size
• pure-llvm-passes-pipe(llvm-functions)
• segregate-stack-accesses(llvm-functions)
• pure-llvm-passes-pipe(llvm-functions)
• emit-function-pointers(llvm-functions)
• savepoint("segregate-stack-accesses", llvm-functions)
  • Artifact: segregate-stack-accesses
  • Analysis: detect-c-strings
  • Analysis: analyze-data-layout

clift-functions branch

Starts from: segregate-stack-accesses

Tasks:

• pure-llvm-passes-pipe(llvm-functions)
• switch-to-statements(llvm-functions)
• pure-llvm-passes-pipe(llvm-functions)
  • Artifact: clifter-input
• fix-pointer-size(llvm-functions)
• clifter(llvm-functions, clift-functions)
  • Artifact: clift-unoptimized
• import-function-data-model(clift-functions)
• pure-mlir-passes-pipe(clift-functions)
• verify-function-against-model(clift-functions)
• savepoint("decompile-clift", clift-functions)
• import-descriptive-function-info(clift-functions)
• savepoint("pre-backend-clift", clift-functions)
  • Artifact: pre-backend-clift

single-file branch

Starts from: clift-functions

Tasks:

• emit-c(clift-functions, decompile-c)
• savepoint("emit-c", decompile-c)
  • Artifact: emit-c
  • Analysis: llm-rename
• emit-c-as-single-file(decompile-c, decompiled-single-file)
• savepoint("emit-c-as-single-file", decompiled-single-file)
  • Artifact: emit-c-as-single-file

helpers branch

Starts from: clift-functions

Tasks:

• emit-helper-header(clift-functions, helper-header)
• savepoint("helper-header", helper-header)
  • Artifact: emit-helper-header

tagless-single-file branch

Starts from: clift-functions

Tasks:

• emit-c(clift-functions, tagless-decompile-c)
• savepoint("emit-tagless-c", tagless-decompile-c)
  • Artifact: emit-tagless-c
• emit-c-as-single-file(tagless-decompile-c, tagless-decompiled-single-file)
• savepoint("emit-tagless-c-as-single-file", tagless-decompiled-single-file)
  • Artifact: emit-tagless-c-as-single-file
• emit-helper-header(clift-functions, tagless-helper-header)
• import-types(clift-types-and-globals)
• import-function-declarations(clift-types-and-globals)
• import-segment-declarations(clift-types-and-globals)
• import-descriptive-info(clift-types-and-globals)
• emit-type-and-global-header(clift-types-and-globals, tagless-type-and-global-header)
• savepoint("tagless-headers", tagless-helper-header, tagless-type-and-global-header)
  • Artifact: emit-tagless-helper-header
  • Artifact: emit-tagless-type-and-global-header
  • Analysis: edit-c-body
• emit-c-as-directory(tagless-decompiled-single-file, tagless-type-and-global-header, tagless-helper-header, recompilable-archive)
• savepoint("recompilable-archive", recompilable-archive)
  • Artifact: emit-recompilable-archive

hexdump branch

Starts from: isolate

Tasks:

• hex-dump(binaries-container, llvm-functions, cfg-map, hexdump)
• savepoint("hexdump", hexdump)
  • Artifact: hexdump

process-assembly branch

Starts from: collect-cfg

Tasks:

• process-assembly(binaries-container, cfg-map, assembly-internal)

disassemble branch

Starts from: process-assembly

Tasks:

• yield-assembly(assembly-internal, assembly)
• savepoint("disassemble", assembly)
  • Artifact: disassemble

recompile branch

Starts from: lift

Tasks:

• link-support(llvm-root)
• pure-llvm-passes-root-pipe(llvm-root)
• compile-root-module(llvm-root, object-file)
• link-for-translation(binaries-container, object-file, translated)
• savepoint("recompile", translated)
  • Artifact: recompile

recompile-isolated branch

Starts from: isolate

Tasks:

• invoke-isolated-functions(llvm-root, llvm-functions, llvm-root-with-functions)
• link-support(llvm-root-with-functions)
• pure-llvm-passes-root-pipe(llvm-root-with-functions)
• compile-root-module(llvm-root-with-functions, object-file)
• link-for-translation(binaries-container, object-file, translated)
• savepoint("recompile-isolated", translated)
  • Artifact: recompile-isolated

cleanup-ir branch

Starts from: segregate-stack-accesses

Tasks:

• emit-string-constants(binaries-container, llvm-functions)
• merge-llvm-modules(llvm-functions, llvm-root-merged)
• pure-llvm-passes-root-pipe(llvm-root-merged)
• savepoint("cleanup-ir", llvm-root-merged)
  • Artifact: cleanup-ir

process-call-graph branch

Starts from: collect-cfg

Tasks:

• process-call-graph(cfg-map, cross-relations)
  • Artifact: cross-relations

yield-call-graph branch

Starts from: process-call-graph

Tasks:

• yield-call-graph(cross-relations, call-graph)
• savepoint("yield-call-graph", call-graph)
  • Artifact: render-svg-call-graph

yield-call-graph-slice branch

Starts from: process-call-graph

Tasks:

• yield-call-graph-slice(cross-relations, call-graph-slice)
• savepoint("yield-call-graph-slice", call-graph-slice)
  • Artifact: render-svg-call-graph-slice

yield-cfg branch

Starts from: process-assembly

Tasks:

• yield-cfg(assembly-internal, function-cfg)
• savepoint("yield-cfg", function-cfg)
  • Artifact: render-svg-cfg

import-types branch

Tasks:

• import-types(clift-types-and-globals)
  • Artifact: import-types
• import-function-declarations(clift-types-and-globals)

emit-type-and-global-header branch

Starts from: import-types

Tasks:

• import-segment-declarations(clift-types-and-globals)
• import-descriptive-info(clift-types-and-globals)
  • Artifact: type-and-global-header-clift
• emit-type-and-global-header(clift-types-and-globals, type-and-global-header)
• savepoint("type-and-global-header", type-and-global-header)
  • Artifact: emit-type-and-global-header

emit-single-type-definition branch

Starts from: import-types

Tasks:

• import-descriptive-info(clift-types-and-globals)
• emit-single-type-definition(clift-types-and-globals, single-type)
• savepoint("single-type-definition", single-type)
  • Artifact: emit-single-type-definition

Analysis lists

• initial-auto-analysis

  • parse-binary
  • import-prototypes-from-db
  • detect-abi
  • detect-stack-size
  • detect-c-strings
  • analyze-data-layout
  • convert-functions-to-cabi