How Syncs work

Before we start, do look at Setting up Knit UI component.

👍
Prerequisites

Read Data Syncs in Knit

Knit API Key

Integration ID for any application whitelisted on Knit that has been successfully validated.

At Knit, syncs are the preferred way for reading data from source applications. Once you create a Knit sync, we will always push data to your endpoint.

In this guide, we will look at the steps needed to setup a sync via Knit:

Prepare your backend to consume Knit sync events.
Start a sync by calling Knit API.

You can use any of the data types mentioned here to set up a sync.

1. Prepare your backend to consume Knit sync events

The very first thing you'd need to do is expose a public API route to receive webhook requests from Knit. This route should only accept POST requests.

The webhook handler needs to handle 5 Knit Events:

record.new

When a new record is encountered eg. a new employee has been added in the HRIS system, or a new application or job has been created in an ATS system

{
  "eventId": "e_123sadasdasda",
  "eventType": "record.new",
  "eventData": {
    // the models that you have subscribed to
  },
  "syncType": "initial_sync",
  "syncDataType": "employee/ats_applications/ats_jobs",
  "syncJobId": "sj_asda32424",
  "syncRunId": "sr_jhhkk80869",
  "recordId": "123"
}

record.modified

When an existing record has been modified eg. an employee's details have been updated in the HRIS system, or an application or job has been modified in the ATS system

{
  "eventId": "e_123sadasdasda",
  "eventType": "record.modified",
  "eventData": {
    // the models that you have subscribed to
  },
  "syncType": "delta_sync",
  "syncDataType": "employee/ats_applications/ats_jobs",
  "syncJobId": "sj_asda32424",
  "syncRunId": "sr_jhhkk80869",
  "recordId": "123"
}

record.deleted

When an exiting record has been deleted or marked as inactive in the application eg. an employee has been marked as inactive in the HRIS system, or an application or job has been deleted in the ATS system

{
  "eventId": "ev_jLPn5iJNhvhDBh91xtb8kc",
  "eventData": {
    // information about which record was deleted
  },
  "eventType": "record.deleted",
  "syncType": "delta_sync",
  "syncDataType": "employee/ats_applications/ats_jobs",
  "syncJobId": "sj_asda32424",
  "syncRunId": "sr_jhhkk80869",
  "recordId": "123"
}

sync.events.processed

You can use this event to audit when a sync has taken place, and the number of records processed

{
  "eventId": "ev_UFrXyD8d5yneJ6UnfxAoJ7",
  "eventData": {
    "status": "processed",
    "processed": 200,
    "emitted": 0,
    "startedAt": "1676292064383",
    "processedAt": "1676292068616",
    "complete": false
  },
  "eventType": "sync.events.processed",
  "syncType": "delta_sync",
  "syncDataType": "employee/ats_applications/ats_jobs",
  "syncJobId": "sj_rJWQawZYVS8IqgtXkSJ8BQ",
  "syncRunId": "sn_agQO87vjVs6tMe9CEsPkBk"
}

sync.events.allConsumed

This event is issued when Knit has finished the sync and all events dispatched have been acknowledged. You can use this event to start any downstream processes.

{
  "eventId": "ev_Z9s8E9mwdwju7dpRcCLh7d",
  "eventData": {
    "status": "completed",
    "processed": 200,
    "emitted": 200,
    "consumed": 200,
    "completedAt": "1676281276296",
    "startedAt": "1676281265551",
    "processedAt": "1676281275876",
    "complete": true
  },
  "eventType": "sync.events.allConsumed",
  "syncType": "initial_sync",
  "syncDataType": "employee/ats_applications/ats_jobs",
  "syncJobId": "sj_rJWQawZYVS8IqgtXkSJ8BQ",
  "syncRunId": "sn_NHXHC2Hn6oOaLrCfsaqGOM"
}

Here's some sample code that shows how to create your API Handler for Knit sync events. But before we begin, do look at Authenticating Knit Webhooks. Make sure to publicly expose this route and pass this as destinationbody parameter in /sync/start API call (see next step below).


//Webhook Handler for Knit Sync, if your server is hosted on https://call.me domain, this api route would be https://call.me/knithook
//You will always receive one employee data as event body on this webhook.  
post("/knithook", (req, res) -> {

  String apiKey = "Your API Key";
  // The integration id for which this sync has been set up.
  String integrationId = req.headers("X-Knit-Integration-Id");
  String actualSignature = req.headers("X-Knit-Signature");
  String reqBody = req.body();
  String expectedSignature = getSignature("Your API Key", reqBody);

  //always validate the signature received in request header.
  if (actualSignature == null ||
      actualSignature.isEmpty() ||
      !actualSignature.equals(expectedSignature)) {
    //block malicious request
    res.status(401);
    return "not ok";
  }

  ObjectMapper mapper = new ObjectMapper();
  JsonNode jsonNode = mapper.readTree(reqBody);
  String eventType = jsonNode.get("eventType").asText();

  switch (eventType) {
    case "record.new":
      // store this record in db and do some downstream processing,
      // like sending a welcome message.
      break;
    case "record.modified":
      // store this record in db and do some downstream processing
      // as per your business rules
      break;
    case "record.deleted":
      // deprovision this employee or application  from your system or handle this event as per your business rules
      break;
    case "sync.events.processed":
      // use this event to keep track of when syncs happen on Knit
      break;
    case "sync.events.allConsumed":
      // use this to start some downstream process once all the data has been synced,
      // like find department wise employee spread, or create org tree for the entire organizatio
      // or calculate job wise analytics etc.
      break;
  }


  // Only 200 status marks consumption of an event,
  // else Knit will retry event delivery
  res.status(200);
  return "ok";
})

public static String getSignature(String key, String data) throws Exception {
  Mac sha256_HMAC = Mac.getInstance("HmacSHA256");
  SecretKeySpec secret_key = new SecretKeySpec(key.getBytes("UTF-8"), "HmacSHA256");
  sha256_HMAC.init(secret_key);
  
  val encoded = Base64
  .getUrlEncoder().withoutPadding()
  .encode(sha256_HMAC.doFinal(data.getBytes(Charsets.UTF_8)))

  return encoded;
}

👍
The webhook will always receive one record data in one request call.
If, for example, you are syncing with some account that has 500 employees, you can expect the webhook to be invoked 500 times with each employee data dispatched individually by Knit. All 500 employees data will not be dispatched together on one API call.

There are two types of syncs that can happen through Knit:

initial_sync: This sync is triggered when a new sync job is configured or when /sync.startAPI is invoked. You can expect active records only in initial_sync, i.e., the event type in payload would be record.new. This is useful if you want to baseline your sync data. Every time /sync.start is invoked, an initial sync will get started.
delta_sync: All sync runs that follow initial_sync are of type delta_sync. In this sync you can expect record.modified and record.deleted events along with record.new.

👍
Test your sync handlers with Knit sandbox
Make sure you test your webhook handler extensively using Knit's sandbox environment before going live on production . Checkout this link for sandbox hostname.

2. Start a sync by calling Knit API

You can start a sync by invoking Start a Sync API

Do look at all the data models that are a part of each sync and subscribe to the ones that you need.

📘
Use the Model ID in models array to subscribe to a data model.

// For example, subscribing to an employee sync with the employee data models
// would look like this:

String payload = "{
     "models": [
          "employee_profile",
          "employee_orgStructure",
          "employee_dependent",
          "employee_location"
     ],
     "frequency": "standard",
     "destination": "https://call.me/knithook", //your webhook
     "dataType": "employee"
}"

//You should have an Integration ID before this step.  
String integrationId = "integration-id-received-from-knit-ui-component";  
Request request = new Request.Builder()
                .url("https://api.getknit.dev/v1.0/sync.start")
                .addHeader("Authorization", "Bearer " + YOUR_KNIT_API_KEY)
                .addHeader("X-Knit-Integration-Id", integrationId)
                .post(RequestBody.create(JSON, payload))
                .build();

Response response = HTTP_CLIENT.newCall(request).execute();
if (!response.isSuccessful()) {
  //handle error
}

//store the sync job id for tracking the sync status and runs.
String syncJobId = jsonMapper.readTree(response.body().string()).get("data").get("syncJobId").asText();

👍
Integration validation webhook call is a good hook for starting syncs.
Use the receipt of integration validation webhook call to start a sync. Do make sure that integration validation call is successful before starting a sync.

Sync frequency can be of three types:

standard: Sync happens once every 24 hrs.
medium: Sync happens once every 12 hrs.
high: Sync happens once very 6 hrs.

You can request for any one of these frequencies while starting a sync subject to your pricing plan.

📘
Start a Sync API is idempotent.
Once a sync job has been setup, on re-invoking this API, you'll receive the same sync job ID. However, every time /sync.start is invoked, an initial sync will get started.
Use Update a Sync API to update the sync job and Pause a Sync API to pause a running sync job.

🚧
Update a Sync API will trigger an initial_sync
Knit will trigger an initial_sync if you update data models' subscription using this API. initial_sync will not be triggered if you just update webhook destination or sync frequency.