Sources
Sources is your tool to pull in data you want to use that already exists. A source create a reference to a table or view that already exists and allows you to use it in your models. They are similar to models, but rather than define them in a .sql
file associated to them, you define the source by specifying the path
to the existing data.
Generating sources
Although we recommend understanding sources, we don't necessarily recommend writing them by hand. It's a time-consuming process and error-prone. In order to make it easier to generate boilerplate code for existing sources, you can use the Visual Studio Code onboarding flow to on onboard sources without having to write the YAML yourself. To do this, open the command palette and run the Quary: Onboarding
command.
A source definition
Sources just like models are defined in project files, for example in models/staging/schema.yaml
. Note here the fully qualified path to the table in the path
property.
Once a source is defined, you can reference it in a model just like you would any other model. For example, the above source can be referenced in a model like below.
In the above example, the fully qualified path to the source contains the project id, dataset id and table id. This is specific to BigQuery and is required to reference a table. If you are using a different database, you may not need to specify the project id and dataset id.
Tests and documentation
Just like models, sources can have tests and documentation. The description
, tags
, and columns
properties work identically.
Equally just like you can reference a model in a SQL test, you can also reference a source in a SQL test with the q.
prefix.