Symbolication¶

Background ¶

The Mozilla Symbolication Server (https://symbolication.services.mozilla.com/) has a symbolication API for converting memory addresses to symbols using debug information generated by builds of Mozilla products that were uploaded to the Mozilla Symbols Server.

Note

2021-10-07

The old symbolication API service hosted on symbols.mozilla.org is deprecated.

Status	URL
Old and deprecated	https://symbols.mozilla.org/symbolicate/v5
New	https://symbolication.services.mozilla.com/symbolicate/v5

Please update to the new service.

Stacks ¶

The stack is a list of lists. Each frame in the stack is composed of two things:

the index in the modules list for the module for this frame
the memory offset relative to the base memory address for that module as an base-10 integer

For example:

[1, 65802]

is the base memory address 65,802 (0x1010a in hex) for the 1-index module in the memoryMap.

Modules ¶

A module is something loaded into memory. It can be a library, binary, font, etc. We need to get the debug information for a given module in order to know what symbols correspond to which addresses.

For Breakpad SYM files, modules are denoted by a debug filename and debug id.

Each debug filename and debug id pair corresponds to a path in the symbols store. The full URL comes from taking the base url, appending the debug filename, appending the debug id, and then appending the debug filename again with a .sym extension.

For example:

["wntdll.pdb", "D74F79EB1F8D4A45ABCD2F476CCABACC2"]

becomes:

https://example.com/v1/wntdll.pdb/D74F79EB1F8D4A45ABCD2F476CCABACC2/wntdll.sym
                       ^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^
                       |          |                                 debug filename with .sym extension
                       |          debug id
                       debug filename

What is Symbolication?¶

Given a set of modules and a set of stacks, symbolication is the act of fetching the debug symbols files for the modules and then looking up the symbols corresponding to the module offsets.

For eaxmple, given this set of modules:

["firefox.pdb", "5F84ACF1D63667F44C4C44205044422E1"],
["mozavcodec.pdb", "9A8AF7836EE6141F4C4C44205044422E1"],
["Windows.Media.pdb", "01B7C51B62E95FD9C8CD73A45B4446C71"],
["xul.pdb", "09F9D7ECF31F60E34C4C44205044422E1"],
...

and this stack:

[3, 6516407],
[3, 12856365],
[3, 12899916],
[3, 13034426],
...

you might end up with something like this:

xul.pdb  mozilla::ConsoleReportCollector::FlushReportsToConsole(unsigned long long, nsIConsoleReportCollector::ReportAction)
xul.pdb  mozilla::net::HttpBaseChannel::MaybeFlushConsoleReports()",
xul.pdb  mozilla::net::HttpChannelChild::OnStopRequest(nsresult const&, mozilla::net::ResourceTimingStructArgs const&, mozilla::net::nsHttpHeaderArray const&, nsTArray<mozilla::net::ConsoleReportCollected> const&)
xul.pdb  std::_Func_impl_no_alloc<`lambda at /builds/worker/checkouts/gecko/netwerk/protocol/http/HttpChannelChild.cpp:1001:11',void>::_Do_call()
...

Tips for Symbolication ¶

If you’re trying to symbolication large numbers of things and it’s going slowly, here are some things you can try:

Always set the User-Agent.

Always set the user agent. If possible, include the url to the git repository where your code is stored.
Wait and retry on HTTP 429 and HTTP 5xx responses.

If you get an HTTP 429 or HTTP 5xx response, wait and retry again. You could use a backoff of (0s, 1s, 2s, 3s, 4s, 5s).

You should always get back a JSON response. If you don’t, treat that like a temporary failure, wait a bit and try again.
If you’re getting a 200 response, but some frames aren’t symbolicated, then either there are no debugging symbols available for that module or the debugging symbols for that module are malformed.

You can see if we have debugging symbols and check to see if they’re well-formed by requesting the module from Mozilla Symbols Server.
Batch symbolication requests as a single request with multiple jobs.

If you batch your symbolication requests it’ll reduce the HTTP request/response overhead and improve the likelihood that the request handler can take advantage of the symcache LRU cache.

Further, it’s best to batch requests for the same platform/buildid/product. That’ll reduce the number of times the request handler has to parse the xul module sym file to at most 1.

For example, if you were trying to symbolicate 1,000,000 crash pings from yesterday, you should:
1. sort them by platform/buildid/product
2. for each platform/buildid/product tuple:
  1. for each N crash pings
    1. create a symbolication request
    2. perform the symbolication request
Currently, a good N is 1,000, but it depends on the size of the stacks and what other modules are involved. You may find a lower N is better.

If this isn’t helpful, we want to know! Please write up a bug and tell us.

Symbolication: /symbolicate/v5 ¶

POST /symbolicate/v5¶

Symbolicate one or more jobs. Each job consists of one or more stacks and a memory map.

Send an HTTP POST request with a JSON payload.

Set a User-Agent. For example:

my-script/1.0 (+https://example.com/path-to-code/)

For documentation on setting useful user agents: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent#crawler_and_bot_ua_strings

Example request:

POST /symbolicate/v5 HTTP/1.1
User-Agent: my-script/1.0 (+https://example.com/path-to-code)

{
  "jobs": [
    {
      "stacks": [
        [
          [
            1,
            7556668
          ],
          [
            1,
            7509754
          ]
        ]
      ],
      "memoryMap": [
        [
          "firefox.pdb",
          "C0A5F7D110D262364C4C44205044422E1"
        ],
        [
          "xul.pdb",
          "0FBE970321AB8CF14C4C44205044422E1"
        ]
      ]
    }
  ],
  "version": 5
}

Example response: [1]

HTTP/1.1 200 OK
Content-Type: application/json

{
  "results": [
    {
      "stacks": [
        [
          {
            "frame": 0,
            "module": "xul.dll",
            "module_offset": "0x734e3c",
            "function": "mozilla::dom::JSActorManager::ReceiveRawMessage(mozilla::dom::JSActorMessageMeta const&, mozilla::Maybe<mozilla::dom::ipc::StructuredCloneData>&&, mozilla::Maybe<mozilla::dom::ipc::StructuredCloneData>&&)",
            "function_offset": "0x7dc",
            "file": "hg:hg.mozilla.org/mozilla-central:dom/ipc/jsactor/JSActorManager.cpp:55c63a6c547f1fecd412505a064f21fd1e1ec48e",
            "line": 172
          },
          {
            "frame": 1,
            "module": "xul.dll",
            "module_offset": "0x7296fa",
            "function": "mozilla::dom::WindowGlobalChild::RecvRawMessage(mozilla::dom::JSActorMessageMeta const&, mozilla::Maybe<mozilla::dom::ClonedMessageData> const&, mozilla::Maybe<mozilla::dom::ClonedMessageData> const&)",
            "function_offset": "0x1ba",
            "file": "hg:hg.mozilla.org/mozilla-central:dom/ipc/WindowGlobalChild.cpp:55c63a6c547f1fecd412505a064f21fd1e1ec48e",
            "line": 586
          }
        ]
      ],
      "found_modules": {
        "firefox.pdb/C0A5F7D110D262364C4C44205044422E1": null,
        "xul.pdb/0FBE970321AB8CF14C4C44205044422E1": true
      }
    }
  ]
}

Here’s an example you can copy and paste–though symbols do age out of our system, so this may not be all that exciting:

curl --user-agent "my-script/1.0 (+https://example.com/)" \
    -d '{"jobs": [{"stacks":[[[0,11723767],[1, 65802]]],"memoryMap":[["xul.pdb","44E4EC8C2F41492B9369D6B9A059577C2"],["wntdll.pdb","D74F79EB1F8D4A45ABCD2F476CCABACC2"]]}]}' \
    https://symbolication.services.mozilla.com/symbolicate/v5

Request JSON Object:

jobs –
array of json objects each specifying a job to symbolicate

[].memoryMap:

array of [debug name (str), debug id (str)] arrays

[].stacks:

array of stacks where each stack is an array of [module index (int), memory offset (int)] arrays

Response JSON Object:

results –
array of result objects–one for every job

[].stacks:

array of symbolicated stacks where each stack is an array of JSON objects

frame (int):

frame index; 0-based

module (str):

the module name

module_offset (str):

the module offset in hex

function (str):

(optional) the function name

function_offset (str):

(optional) the function offset in hex

file (str):

(optional) the source file

line (int):

(optional) the line number in the source file

inlines (array):

(optional) an array of inline frames, each of which has optional “function”, “file” and “line” properties, ordered from deepest call to most shallow

[].found_modules:

json object indicating which modules we had symbols for and which ones we didn’t

<debug_filename>/<debug_id> (str):

true if we found symbols, false if we didn’t, and null if we didn’t need to look up symbols because it’s not referenced in the stacks

Request Headers:

User-Agent –
please provide a unique user agent to make it easier for us to help you debug problems

For example:
```
my-script/1.0 (+https://example.com/)
```
See the MDN docs on good user agent practices:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent#crawler_and_bot_ua_strings
Debug – if you add Debug: true to the headers, then symbolication will also return debug information about cache lookups, how many downloads, timings, and some other things

Status Codes:

200 OK – success symbolicating stacks
429 Too Many Requests – your request was rate-limited; wait a bit and retry [2]
500 Internal Server Error – something bad happened–please open up a bug
503 Service Unavailable – temporary problem with the server; wait a bit and retry

Symbolication: /symbolicate/v4 (deprecated)¶

POST /symbolicate/v4¶

Symbolicate one or more stacks.

Send an HTTP POST request with a JSON payload.

Warning

Don’t use Symbolication v4 for anything new. Please migrate to v5 as soon as you can. v4 will be removed in the near future.

Follow this bug for status:

https://bugzilla.mozilla.org/show_bug.cgi?id=1475334

Example request:

POST /symbolicate/v4 HTTP/1.1

{
  "memoryMap": [
    [
      "xul.pdb",
      "44E4EC8C2F41492B9369D6B9A059577C2"
    ],
    [
      "wntdll.pdb",
      "D74F79EB1F8D4A45ABCD2F476CCABACC2"
    ]
  ],
  "stacks": [
    [
      [0, 11723767],
      [1, 65802]
    ]
  ],
  "version": 4
}

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "symbolicatedStacks": [
    [
      "XREMain::XRE_mainRun() (in xul.pdb)",
      "KiUserCallbackDispatcher (in wntdll.pdb)"
    ]
  ],
  "knownModules": [
    true,
    true
  ]
}

Symbolication¶

Background¶

Stacks¶

Modules¶