Graphing Xcode project dependencies — Introducing XCGrapher
Recently we’ve been migrating our (many) in-house libraries from CocoaPods to Swift Packages. To get an overview of our progress I built a new tool and subsequently learnt a bunch of lessons in something new to me: dynamically loading Swift libraries at runtime!
This is a bit of a longer article, so to whet your appetite here’s a partial list of the things we’ll cover (sometimes in excruciating detail) :
- 💡 Why I created
xcgrapher
and how it works - 👨💻 How to write an
xcgrapher
plugin - 👨🔧 How to stop the Swift Compiler from mangling function names
- 🧙♂️ How to load and execute code in an external dynamic library at runtime
- 🍺 How to distribute a Swift Package via Homebrew
- 🕵️♂️ How to see and change dynamic library paths embedded in a binary
Search for the emoji above to jump to a section!
💡Module Migration Mayhem
Moving 10+ custom CocoaPods to Swift Packages is a tricky task — especially so when they are inter-dependent and are all used by three iOS/tvOS app projects. Deciding the order of migration and determining which dependencies will be affected quickly became difficult, so having a graph to show who is dependent on whom sounded like a very good idea.
There seems to be various dependency graphing tools already available however none of them fulfilled all my requirements. For my purposes, a graphing tool must:
- Graph at a module-level, not class-level
- Support private/in-house libraries
- Show both Swift Packages and CocoaPods in a single graph
Finding Imported Modules
Initially I experimented with objdump
(more on this later) as a way of showing what libraries a binary is dependent on, however it quickly became apparent that not all cases could be covered this way. As each module import is represented by a line of code such as import MyModule
… it seemed the course of action would be to simply let the tool read these imports.
Reading an Xcode target’s Compile Sources list
If you’ve worked on an Xcode project with a team of any size you’ve no doubt had to deal with the headache of reading the project.pbxproj
file when resolving merge conflicts. I had zero desire to spend time deciphering it’s format just to get a list of compile sources so luckily the Cocoapods team have a tool for interacting with it.
require 'xcodeproj'Xcodeproj::Project.open("SomeApp.xcodeproj")
.targets
.filter do |target|
target.name == "SomeApp"
end
.first
.source_build_phase.files.to_a
.each do |file|
puts file.file_ref.real_path.to_s
end
This script… is in Ruby. Which is annoying, because Ruby means Gems and Gems means Bundler and Bundler means troubleshooting an Rbenv setup far too often. But… most iOS engineers will have done all that already for CocoaPods and it’s better than parsing a pbxproj, so I compressed it into a shell-able one-liner and that what is currently being used in xcgrapher
:
$ ruby -r xcodeproj -e 'Xcodeproj::Project.open("XXX").targets.filter do |t| t.name == "YYY" end.first.source_build_phase.files.to_a.each do |f| puts f.file_ref.real_path.to_s end'
Moving on now.
Finding imports
This was relatively straight forward: loop through the compile sources list, find all the lines with import Something
and compile all the Something
s into a list. There are many variants on importing, including:
import Module
@testable import Module
import class Module.Class
@_exported import Module
I accounted for as many of these as I could think of — and then some. Take a look at the ImportFinder for the full list.
Find imports in Swift Packages
The same ImportFinder
code then needed to be run on every Swift Package. As far as I could tell there was no easy way to list all Swift Packages that an Xcode project is dependent on and because I also didn’t want my tool to be responsible for having to clone the correct commit of each package before reading it’s source I opted for simply running xcodebuild and parsing it’s output to find local package clones.
Once each package clone directory is known, getting a list of source files for each target is easy!
$ swift package -—package-path /path/to/clone describe -—type json
And then the ImportFinder
mentioned earlier can be used to read all the imports in each package product.
Finding imports in Cocoapods
This was less of a priority for me as we are moving away from CocoaPods — in fact, xcgrapher
is (currently) mostly geared towards projects that just use Swift Packages. The easiest way to compile a list of known CocoaPods was to parse the Podfile.lock
: if you’re interested in that then see the implementation here.
Graphing with GraphViz
Once a list of all imported modules is compiled then xcgrapher
iterates through them and matches them up with other modules. The actual graphing is done through GraphViz using a digraph. A digraph is a format for specifying “edges” (arrows in a directed-graph) between “nodes” on a graph:
digraph XCGrapher {
graph [ nodesep = 0.5, ranksep = 4, overlap = false, splines = true ]
node [ shape = box ]
"Algorithms" -> "RealModule"
"Charts" -> "Algorithms"
"RealModule" -> "_NumericsShims"
"RealmSwift" -> "Realm"
"SomeApp" -> "Charts"
"SomeApp" -> "Lottie"
"SomeApp" -> "RealmSwift"
}
If you want to try it out yourself simply brew install graphviz
and then dot -T png -o output.png digraph.txt
to generate a PNG image from a text file containing the above. When running xcgrapher
locally your digraph is stored at /tmp/xcgrapher.dot
.
This is the simplest of examples. GraphViz is capable of so, so much more. Do check it out!
👨💻 Creating Custom Graphs with XCGrapher Plugins
We were only getting started.
What if you want to graph something else? For example, our projects use a custom property wrapper @Injected
as a way of injecting dependencies into class instances. A graph of which classes require which injections could also be useful in the future…
Rather than writing all the boiler plate described above, you can make use of an xcgrapher
plugin for these custom tasks.
Plugin structure
xcgrapher
plugins are just Swift Packages that compile to dynamic libraries and are dependent on XCGrapherPluginSupport
:
let package = Package(
name: "MyPlugin",
products: [
.library(
name: "MyPlugin",
type: .dynamic,
targets: ["MyPlugin"]
),
],
dependencies: [
.package(
url: "https://github.com/maxchuquimia/XCGrapherPluginSupport.git",
from: "0.0.6"
),
],
targets: [
.target(
name: "MyPlugin",
dependencies: [
.product(
name: "XCGrapherPluginSupport",
package: "xcgrapher"
)
]
),
]
)
Within the package you can import XCGrapherPluginSupport
and subclass XCGrapherPlugin
. Then you can override functions to help build up your graph:
override func process(file: XCGrapherFile) throws -> [Any] { ... }
This function is called once for every source file in your project and once for every source file in your Swift Package dependencies (if --spm
is passed to xcgrapher
). The model XCGrapherFile
contains convenient info about the source file that you can then parse in your own way (see XCGrapherFile).
override func process(library: XCGrapherImport) throws -> [Any] { ... }
This function is called once for every import SomeLibrary
line in your projects (depending on the flags passed to xcgrapher
). SomeLibrary
is represented by the XCGrapherImport
model. See XCGrapherImport for available parameters.
override func makeArrows(from results: [Any]) throws -> [XCGrapherArrow]
The arrays returned by the processing functions listed above will be concatenated and passed to you for final consolidation in the makeArrows(from:)
function. In here you should loop through results
as you see fit so as to return a list of XCGrapherArrow
models that will be passed to GraphViz and drawn to xcgrapher
's --output
PNG file.
I chose to use Any
here to give as much flexibility as possible to you, the plugin builder. Forcing plugins to be subclasses of XCGrapherPlugin
rather than any object that conforms to a protocol was another strategic decision to make the plugin builder’s life easier as some Dodgy Things™ need to be done later — keep reading!
So, if you wanted to graph all @Injected
lines in every file (including Swift Packages) all you need to do is parse file
in process(file:)
, return your own object representing the @Injected
relationship and then convert that object into an XCGrapherArrow
in the makeArrows(from:)
function!
👨🔧 Avoiding mangled function names
Have you every po
-ed something in Xcode’s debugger and noticed that a class’s name looks… mangled? For example, consider the following Swift file:
// doSomething.swiftpublic func thisFunctionDoesSomething() {
print("doing something...")
}
If you were to compile this doSomething.swift
into a dynamic library:
$ swiftc -emit-library -module-name libDoSomething doSomething.swift
And then use nm
print out all the symbols within the resulting dylib, you can see that the name of our function thisFunctionDoesSomething()
is no longer what we would expect:
$ nm -an liblibDoSomething.dylib
U _$sSS21_builtinStringLiteral17utf8CodeUnitCount7isASCIISSBp_BwBi1_tcfC
U _$sSSN
U _$ss27_allocateUninitializedArrayySayxG_BptBwlF
U _$ss5print_9separator10terminatoryypd_S2StF
U _$sypN
U _swift_bridgeObjectRelease
U dyld_stub_binder
0000000000003e50 T _$s14libDoSomething016thisFunctionDoesC0yyF
0000000000003f00 t _$ss5print_9separator10terminatoryypd_S2StFfA0_
0000000000003f20 t _$ss5print_9separator10terminatoryypd_S2StFfA1_
0000000000003fa8 s ___swift_reflection_version
0000000000008020 d __dyld_private
The Swift compiler emits mangled names into binary images to encode references to types for runtime instantiation and reflection. In a binary, these mangled names may embed pointers to runtime data structures in order to more efficiently represent locally-defined types.
— https://github.com/apple/swift/blob/main/docs/ABI/Mangling.rst
In a nutshell, name mangling lets the compiler generate a new name for a function in a deterministic way — this is important because it allows us developers to write (for example) two functions with the same name (something we do every time we override a function in a subclass).
To avoid mangling in Swift we can use the @_cdecl
attribute to specify the exact symbol name that the compiler should store for the function being annotated:
// doSomething.swift@_cdecl("thisFunctionDoesSomething")
public func thisFunctionDoesSomething() {
print("doing something...")
}
Now when we compile and list the symbols in the dylib binary we can see our function name clearly!
$ swiftc -emit-library -module-name libDoSomething doSomething.swift
$ nm -an liblibDoSomething.dylib
U _$sSS21_builtinStringLiteral17utf8CodeUnitCount7isASCIISSBp_BwBi1_tcfC
U _$sSSN
U _$ss27_allocateUninitializedArrayySayxG_BptBwlF
U _$ss5print_9separator10terminatoryypd_S2StF
U _$sypN
U _swift_bridgeObjectRelease
U dyld_stub_binder
0000000000003e40 T _thisFunctionDoesSomething
0000000000003e50 T _$s14libDoSomething016thisFunctionDoesC0yyF
0000000000003f00 t _$ss5print_9separator10terminatoryypd_S2StFfA0_
0000000000003f20 t _$ss5print_9separator10terminatoryypd_S2StFfA1_
0000000000003fa8 s ___swift_reflection_version
0000000000008020 d __dyld_private
When loading a library at runtime we’ll need to know the names of symbols before they are compiled so that we can call them — so annotating a function in the plugin dylib that creates the custom XCGrapherPlugin
subclass is a must:
@_cdecl("makeXCGrapherPlugin")
public func makeXCGrapherPlugin() -> UnsafeMutableRawPointer {
Unmanaged.passRetained(MyPlugin()).toOpaque() }
We use an UnsafeMutableRawPointer
here because we are doing Dodgy Things™ and the Swift compiler wants no responsibility for the multitude of things that could go wrong.
that’s Swift stepping back with its hands up in the air… “hey, you’re on your own, Bud”
— My colleague after I showed him the implementation
(actually @_cdecl
marks the function as a C declaration so we need to use a C-compatible return value. There was some talk around making it officially supported, hopefully it happens soon!)
Compiling the dynamic library
Build the library the same way you build any Swift Package:
$ swift build -c release --disable-sandbox
The library is then saved to ./.build/release/libMyPlugin.dylib
🧙♂️ Loading a dynamic library at runtime
Now that we have our dylib stored on the disk, back in the main code we can pass it’s path to dlopen()
and retrieve the symbol that points to the function marked with @_cdecl("makeXCGrapherPlugin")
from earlier:
let path = "/path/to/libMyPlugin.dylib"
let openResult = dlopen(path, RTLD_NOW|RTLD_LOCAL)
guard openResult != nil else { ... }
defer { dlclose(openResult) }let symbol = dlsym(openResult, "makeXCGrapherPlugin")
If you want to know about RTLD_NOW
and RTLD_LOCAL
(I know you don’t!), then here’s some info for you. I seem to remember reading that RTLD_LOCAL
may not actually be needed because it’s default unless you specify RTLD_GLOBAL
… but half the Swift examples I’ve seen online use it so I’m leaving it in for now. More about me not knowing if I should remove things later.
ANYWAY, then we can cast that symbol to a function with the same signature as the Dodgy Things™ function in the dynamic library:
typealias MakeFunc = @convention(c) () -> UnsafeMutableRawPointerlet makePlugin = unsafeBitCast(symbol, to: MakeFunc.self)
And finally call the function to get an instance of the XCGrapherPlugin
subclass defined in the dynamic library!
let pluginPointer = makeXCGrapherPlugin()
let plugin = Unmanaged<XCGrapherPlugin>.fromOpaque(pluginPointer)
.takeRetainedValue()
yeah wow. that’s a lot simpler than I expected
— My colleague (the same colleague)
To use your newly created xcgrapher
plugin, pass it’s path to the --plugin
option of the xcgrapher
tool.
Live examples & references
xcgrapher
’s dynamic library loading code can be viewed here. The default behaviour of graphic module imports is implemented as a standalone plugin and can be viewed here.
I first learnt about loading libraries in this way from this fantastic article — their approach is a little different to mine so do have a read if you want to learn more! Finally, here’s some official literature on dlopen()
and friends.
Publishing
The final step of building a tool is making it easily installable. In the macOS world, this mean making it installable via brew
.
If you want to install xcgrapher
you’ll need to add my tap first.
$ brew tap maxchuquimia/scripts
$ brew install xcgrapher
# If you don’t use CocoaPods you’ll also need to install Xcodeproj.
I chose to use my own tap for this rather than adding it to the main brew tap because I’m not really a fan of keeping everything centralized in the way that Homebrew and CocoaPods do (for speed reasons). I personally hope Swift Packages never go this way…
Compilation, installation and embedded dynamic library paths
Swift Packages that produce executables are really no different to any brew-installable tool: just build the binary and move it to the location specified by brew
. I used a Makefile to do this:
prefix ?= /usr/local
bindir = $(prefix)/bin
libdir = $(prefix)/lib
buildroot = $(shell swift build -c release --show-bin-path)
configure:
echo "let DEFAULT_PLUGIN_LOCATION=\"$(libdir)/libXCGrapherModuleImportPlugin.dylib\"" > Sources/xcgrapher/Generated.swift
build: configure
swift build -c release --disable-sandbox
install: build
mkdir -p "$(bindir)"
mkdir -p "$(libdir)"
install "$(buildroot)/xcgrapher" "$(bindir)"
install "$(buildroot)/libXCGrapherPluginSupport.dylib" "$(libdir)"
install "$(buildroot)/libXCGrapherModuleImportPlugin.dylib" "$(libdir)"
install_name_tool -change "$(buildroot)/libXCGrapherPluginSupport.dylib" "$(libdir)/libXCGrapherPluginSupport.dylib" "$(bindir)/xcgrapher"
uninstall:
rm -rf "$(bindir)/xcgrapher"
rm -rf "$(libdir)/libXCGrapherPluginSupport.dylib"
rm -rf "$(libdir)/libXCGrapherModuleImportPlugin.dylib"
clean:
rm -rf .build
rm Sources/xcgrapher/Generated.swift
.PHONY: build install uninstall clean configure
There’s a few things going on here! First up there are a few definitions, then a few steps:
prefix ?= /usr/local
This defines the variable prefix
to be /usr/local
only if a prefix
hasn’t already been defined when calling make
(more on this later)
bindir and libdir
These two variables define the directory in which binaries and libraries should be installed.
buildroot
This variable is set to the output of swift build -c release --show-bin-path
— which is the directory where swift build
will write artifacts when compiling.
configure:
The configure
target writes the predicted dylib installation path into a file within xcgrapher
’s package. This then gets picked up during compile time and the generated DEFAULT_PLUGIN_LOCATION
variable is used as a default value for the --plugin
argument.
build:
The build
target first runs the configure step. Then it ensures the binary and library directories exist using mkdir -p
. The first install
commands safely copies the xcgrapher
binary to the binary directory and the second and third install
commands copy libXCGrapherPluginSupport.dylib
and libXCGrapherModuleImportPlugin.dylib
to the library directory.
🕵️♂️ install_name_tool
install_name_tool
is an interesting command: it opens a binary (the third argument) and changes dylib paths matching the first argument to the path that is the second argument. Using install_name_tool
seems to be recommended here by Matt Thompson (a reliable source of knowledge) however in his article he doesn’t really go into detail about in which cases it is needed. So, of course I decided to investigate.
Let’s take a look at all the dynamic libraries that xcgrapher
depends on (note that /usr/local/bin/xcgrapher
is the path to the xcgrapher
binary installation) :
$ objdump -p /usr/local/bin/xcgrapher
/usr/local/bin/xcgrapher: file format Mach-O 64-bit x86-64
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
MH_MAGIC_64 X86_64 ALL 0x00 EXECUTE 30 4240 NOUNDEFS DYLDLINK TWOLEVEL PIE
Load command 0
<redacted>
<redacted>
<redacted>
Load command 13
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libobjc.A.dylib (offset 24)
time stamp 2 Thu Jan 1 10:00:02 1970
current version 228.0.0
compatibility version 1.0.0
Load command 14
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan 1 10:00:02 1970
current version 1292.60.1
compatibility version 1.0.0
Load command 15
cmd LC_LOAD_DYLIB
cmdsize 64
name @rpath/libXCGrapherPluginSupport.dylib (offset 24)
time stamp 2 Thu Jan 1 10:00:02 1970
current version 0.0.0
compatibility version 0.0.0
Load command 16
cmd LC_LOAD_DYLIB
cmdsize 96
name /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (offset 24)
time stamp 2 Thu Jan 1 10:00:02 1970
current version 1770.255.0
<redacted>
<redacted>
<redacted>
Okay let’s try that again, this time with some filtering (only printing lines matching LC_LOAD_DYLIB
and the two after it) :
$ objdump -p /usr/local/bin/xcgrapher | grep LC_LOAD_DYLIB -A 2
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libobjc.A.dylib (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 64
name @rpath/libXCGrapherPluginSupport.dylib (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 96
name /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 64
name /usr/lib/swift/libswiftCore.dylib (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 64
name /usr/lib/swift/libswiftDarwin.dylib (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 64
name /usr/lib/swift/libswiftFoundation.dylib (offset 24)
Here you can clearly see all the dylibs that xcgrapher
requires — including our very own libXCGrapherPluginSupport
(note that this isn’t the plugin itself as we load that at runtime: this is the package that defines the superclass XCGrapherPlugin
that we spoke about earlier).
libXCGrapherPluginSupport
isn’t given a full path like the rest of the libraries: instead it starts with @rpath
! This is a special macro that allows us to avoid having to do things like use install_name_tool
and enable us developers to distribute programs with dylibs along side them. In fact, the install_name_tool
command in the Makefile seems entirely useless as it tries to match .build/release/libXCGrapherPluginSupport.dylib
instead of @rpath/libXCGrapherPluginSupport.dylib
.
The real way to change the path of the dylib that should be loaded is:
$ install_name_tool -change "@rpath/libXCGrapherPluginSupport.dylib" "/new/dylib/path/something.dylib" /usr/local/bin/xcgrapher
And now when we run the search for loaded libraries again:
$ objdump -p /usr/local/bin/xcgrapher | grep LC_LOAD_DYLIB -A 2
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libobjc.A.dylib (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 56
name /new/dylib/path/something.dylib (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 96
name /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 64
name /usr/lib/swift/libswiftCore.dylib (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 64
name /usr/lib/swift/libswiftDarwin.dylib (offset 24)
--
cmd LC_LOAD_DYLIB
cmdsize 64
name /usr/lib/swift/libswiftFoundation.dylib (offset 24)
libXCGrapherPluginSupport.dylib
is no where to be seen. If we try to run xcgrapher
now we get an error that may be familiar to some of us:
$ xcgrapher
dyld: Library not loaded: /new/dylib/path/something.dylib
Referenced from: /usr/local/bin/xcgrapher
Reason: image not found
Abort trap: 6
The reason for me leaving install_name_tool
in my Makefile is mainly concern around my lack of knowledge on the subject. Perhaps Matt knows something I don’t know, perhaps on his machine when he runs swift build
then the @rpath
macro isn’t used? I only have two computers here to test installing on, maybe if I tried it on a third then it would be needed? For now, I’m keeping it.
Anyway, back to walking through the Makefile.
uninstall:
The uninstall
target removes the binaries and libraries from your machine.
clean:
The clean
target removes any artifacts that were created by the build
step.
.PHONY
This defines a list of targets that aren’t files (from what I can gather make
is really all about creating files so we need to ensure it doesn’t get confused by what we’ve written — more about that here).
🍺 Homebrew Formulae
The brew
command reads a “Formula” that defines what needs to be done in order for a command to be installed. xcgrapher
’s formula looks like this:
class Xcgrapher < Formula
desc "Framework-level dependency graph generator for Xcode projects "
homepage "https://github.com/maxchuquimia/xcgrapher"
url "https://github.com/maxchuquimia/xcgrapher.git",
:tag => "0.0.8", :revision => "d5b410f1f42bb95ce51453da539834d3678d85a4"
head "https://github.com/maxchuquimia/xcgrapher.git"
depends_on :xcode => ["12.4", :build]
depends_on "graphviz"
def install
system "make", "install", "prefix=#{prefix}"
end
end
First, information about the source code and version is defined. Then depends_on
ensures Xcode exists and installs graphviz
if necessary (remember, xcgrapher
needs GraphViz’ dot
command). Then the install
function calls make install
in the directory of the project clone.
As mentioned earlier in this article, prefix=#{prefix}
is where the prefix
variable in the Makefile is overwritten. By using the formula’s prefix instead of the constant /usr/local
we wrote in the Makefile we allow Homebrew to manage the installation gracefully. In my case prefix
turned out to be /usr/local/Cellar/xcgrapher/0.0.8
. But don’t worry, brew symbolically links everything to /usr/local
anyway (that’s why I could use /usr/local/bin/xcgrapher
in my earlier examples).
xcgrapher
’s formula is published in my tap. The simplest custom tap is a GitHub repo under your name called hombrew-scripts
with formula stored in a Formula
directory. Of course, it gets more involved in some cases: read all about it here.
dlclose(thisArticle)
Thanks for reading this far! I hope xcgrapher
is useful to you in some way, even if it’s just a starting point for something entirely different.
I’ll leave you with xcgrapher --help
— but as for now, goodnight :)