> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oleria.com/llms.txt
> Use this file to discover all available pages before exploring further.

# File-Based Integration

Oleria's Custom Application integration lets you connect any custom data source - an internal app, an HR system, a database - to Oleria without a native integration. You export your data as CSV files, upload them to an S3 bucket you control, and Oleria automatically reads, maps, and syncs the data on a recurring schedule.

## How it works

<img src="https://mintcdn.com/oleria/9eNE4qr_2dvBWbfD/images/integrations/custom-application/how_it_works_combined.png?fit=max&auto=format&n=9eNE4qr_2dvBWbfD&q=85&s=7a6c5a34a8442f5c014ae754fe24720a" alt="How the Custom Application integration works" width="900" height="566" data-path="images/integrations/custom-application/how_it_works_combined.png" />

## Prerequisites

* An Amazon S3 bucket in your AWS account where you will upload your data.
* AWS IAM admin access to add a bucket policy to that S3 bucket.
* Your application's data exported as CSV files (comma-separated values with a header row).

<Note>Use a dedicated S3 bucket or a scoped prefix within your bucket for Oleria data to limit access to only what is needed.</Note>

## Set Up Your S3 Bucket

<Steps>
  <Step title="Create or identify your S3 bucket">
    Create or identify the S3 bucket you will use to share data with Oleria.
  </Step>

  <Step title="Organize your files">
    Organize your files inside the bucket using the following folder structure:

    <img src="https://mintcdn.com/oleria/9eNE4qr_2dvBWbfD/images/integrations/custom-application/s3_folder_structure.png?fit=max&auto=format&n=9eNE4qr_2dvBWbfD&q=85&s=11b70b03550573c008473931e6052e96" alt="S3 bucket folder structure" width="800" height="500" data-path="images/integrations/custom-application/s3_folder_structure.png" />

    <Note>
      The <code>config/</code> folder must contain exactly one <code>.yaml</code> file - Oleria discovers it automatically. The date-stamped folder under <code>rawdata/</code> (e.g. <code>startedat=2026-04-17T00:00:00Z</code>) tells Oleria which data is new. Each time you export fresh data, create a new folder with the current timestamp. Oleria always picks the most recent one. If your data does not change frequently, you can place your CSV subfolders directly inside <code>rawdata/</code> without a date-stamped folder.
    </Note>
  </Step>
</Steps>

## Prepare Your CSV Files

Your CSV files must have a header row. The column names in the header are what you reference in the `config.yaml` file. Column names can be anything - you map them to Oleria's fields in the config.

**Example `users.csv`:**

<div style={{overflowX: 'auto'}}>
  | email                                         | full\_name | username | status | created\_at | department  | title             |
  | --------------------------------------------- | ---------- | -------- | ------ | ----------- | ----------- | ----------------- |
  | [alice@example.com](mailto:alice@example.com) | Alice Chen | achen    | active | 2023-01-15  | Engineering | Software Engineer |
  | [bob@example.com](mailto:bob@example.com)     | Bob Smith  | bsmith   | active | 2022-06-01  | Sales       | Account Executive |
</div>

**Example `roles.csv`:**

<div style={{overflowX: 'auto'}}>
  | role\_id | role\_name    | description        |
  | -------- | ------------- | ------------------ |
  | admin    | Administrator | Full system access |
  | viewer   | Read Only     | View-only access   |
</div>

**Example `user_roles.csv`** (membership - which user has which role):

<div style={{overflowX: 'auto'}}>
  | user\_email                                   | role\_id |
  | --------------------------------------------- | -------- |
  | [alice@example.com](mailto:alice@example.com) | admin    |
  | [bob@example.com](mailto:bob@example.com)     | viewer   |
</div>

## Create Your config.yaml

The `config/config.yaml` file maps your CSV columns to Oleria's data model. Your Oleria Customer Success team can provide a starting config based on your data, or you can write it yourself using the format below.

<Note>Oleria will soon provide an automated way to analyze your CSV data and generate the <code>config.yaml</code> for you - no manual authoring required. Coming soon.</Note>

#### What object types can I map to?

**Entity objects** - use these for things like users, roles, and groups:

| Object type            | Use when your CSV contains...                                     |
| ---------------------- | ----------------------------------------------------------------- |
| `applicationaccount`   | Application users, login accounts, or service accounts            |
| `identity`             | Users from an Identity Provider (Okta, PingOne, Active Directory) |
| `role`                 | Roles, profiles, or permission sets                               |
| `usergroup`            | Groups, teams, or security groups                                 |
| `person`               | Person records from an HR system (the human behind the accounts)  |
| `employee`             | Employee HR data - job title, manager, department, start date     |
| `department`           | Department or org unit records                                    |
| `activity_analysis`    | Audit log events - one row per event                              |
| `application_instance` | Applications managed by an IdP (e.g. apps in your Okta catalog)   |

**Relationship objects** - connect two entities together. Always use the `links` shorthand for these:

| Use this value for `obs_object_name`               | What it connects                     |
| -------------------------------------------------- | ------------------------------------ |
| `applicationaccount_memberof_role`                 | Application user → Role              |
| `applicationaccount_memberof_usergroup`            | Application user → Group             |
| `identity_memberof_role`                           | IdP user → Role                      |
| `identity_memberof_usergroup`                      | IdP user → Group                     |
| `identity_assigned_accessto_application_instance`  | IdP user → Application               |
| `usergroup_assigned_accessto_application_instance` | Group → Application                  |
| `usergroup_memberof_role`                          | Group → Role                         |
| `person_is_employee`                               | Person → Employee record             |
| `employee_manages_employee`                        | Manager → Employee                   |
| `employee_serving_in_department`                   | Employee → Department                |
| `department_contains_department`                   | Parent department → Child department |

#### `source.s3` fields

These fields go inside `source.s3` in your `config.yaml`. They tell Oleria where your S3 data lives and how frequently to sync access data and activity logs.

| Field                          | Required | Default | Description                                                                                                                             |
| ------------------------------ | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `partition_prefix`             | Yes      | -       | Prefix for time-stamped folders under `rawdata/` (e.g. `"startedat="`). Use `""` if CSVs are placed directly inside `rawdata/`.         |
| `rbac_sync_interval_hours`     | No       | `3`     | How often (in hours) you drop new access data (users, roles, groups) into S3. Oleria schedules the RBAC sync pipeline at this interval. |
| `activity_sync_interval_hours` | No       | `1`     | How often (in hours) you drop new activity/audit-log data into S3. Oleria schedules the activity sync pipeline independently from RBAC. |

#### `field_mappings` vs `links`

| Key              | When to use                                                                                        |
| ---------------- | -------------------------------------------------------------------------------------------------- |
| `field_mappings` | Use for **entity objects** - maps CSV columns to Oleria fields (`Name`, `Email`, `Title`, etc.)    |
| `links`          | Use for **relationship objects** - connects two entities by referencing the ID column of each side |

<Accordion title="Available fields reference">
  **applicationaccount** and **identity** - Application & IdP users

  | Field                         | Description                    | Notes                                                                                                                                   |
  | ----------------------------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
  | <code>Name</code>             | Display name                   |                                                                                                                                         |
  | <code>Email</code>            | Email address                  |                                                                                                                                         |
  | <code>Alias</code>            | Username or login handle       |                                                                                                                                         |
  | <code>Enabled</code>          | Active status                  | Map to <code>true</code> / <code>false</code>                                                                                           |
  | <code>CreatedDate</code>      | Account creation date          | Date auto-detected                                                                                                                      |
  | <code>LastActivityDate</code> | Date of last recorded activity | Date auto-detected                                                                                                                      |
  | <code>Title</code>            | Job title                      |                                                                                                                                         |
  | <code>Department</code>       | Department or team name        |                                                                                                                                         |
  | <code>CompanyName</code>      | Company or organization name   |                                                                                                                                         |
  | <code>IsAdmin</code>          | Admin privileges               | Map to <code>true</code> / <code>false</code>                                                                                           |
  | <code>SubType</code>          | Account sub-type               | Default: <code>StandardUser</code>. Options: <code>StandardUser</code>, <code>Application</code>, <code>Machine</code>                  |
  | <code>Type\_</code>           | Account type                   | Default: <code>User</code>. Options: <code>User</code>, <code>Machine</code>                                                            |
  | <code>MfaRequirements</code>  | MFA requirement status         | Default: <code>Unavailable</code>. Options: <code>Unavailable</code>, <code>Required</code>, <code>NotApplicable</code>                 |
  | <code>SsoRequirements</code>  | SSO requirement status         | Default: <code>Unavailable</code>. Options: <code>Unavailable</code>, <code>Required</code>, <code>NotApplicable</code>                 |
  | <code>LicenseLevel</code>     | License tier                   | Default: <code>Unavailable</code>. Options: <code>Unavailable</code>, <code>Free</code>, <code>Standard</code>, <code>Enterprise</code> |
  | <code>LicenseStatus</code>    | License status                 | Default: <code>NotApplicable</code>. Options: <code>NotApplicable</code>, <code>Active</code>, <code>Expired</code>                     |
  | <code>SourceTag</code>        | Label for this data source     | e.g. <code>postgres-users</code>                                                                                                        |

  ***

  **role** - Roles & permission sets

  | Field                    | Description                    | Notes                                                                               |
  | ------------------------ | ------------------------------ | ----------------------------------------------------------------------------------- |
  | <code>Name</code>        | Display name of the role       |                                                                                     |
  | <code>Description</code> | What the role grants or allows |                                                                                     |
  | <code>Type\_</code>      | Role type                      | Default: <code>Standard</code>. Options: <code>Standard</code>, <code>Custom</code> |
  | <code>IsCustom</code>    | Custom-defined role            | Map to <code>true</code> / <code>false</code>                                       |
  | <code>CreatedDate</code> | Date the role was created      | Date auto-detected                                                                  |
  | <code>SourceTag</code>   | Label for this data source     |                                                                                     |

  ***

  **usergroup** - Groups & teams

  | Field                    | Description                | Notes                                                                                                |
  | ------------------------ | -------------------------- | ---------------------------------------------------------------------------------------------------- |
  | <code>Name</code>        | Display name of the group  |                                                                                                      |
  | <code>Description</code> | What the group is used for |                                                                                                      |
  | <code>Type\_</code>      | Group type                 | Default: <code>Custom</code>. Options: <code>Custom</code>, <code>Built-in</code>, <code>Sync</code> |
  | <code>CreatedDate</code> | Date the group was created | Date auto-detected                                                                                   |
  | <code>SourceTag</code>   | Label for this data source |                                                                                                      |

  ***

  **person** - HR person records

  | Field                             | Description                 | Notes              |
  | --------------------------------- | --------------------------- | ------------------ |
  | <code>Name</code>                 | Full name of the person     |                    |
  | <code>PrimaryPersonalEmail</code> | Personal email address      |                    |
  | <code>CreatedDate</code>          | Date the record was created | Date auto-detected |

  ***

  **employee** - HR employee records

  | Field                         | Description                     | Notes                       |
  | ----------------------------- | ------------------------------- | --------------------------- |
  | <code>WorkName</code>         | Full name as used at work       |                             |
  | <code>PrimaryWorkEmail</code> | Work email address              |                             |
  | <code>EmployeeNumber</code>   | Employee ID from your HR system |                             |
  | <code>StartDate</code>        | Employment start date           | Date auto-detected          |
  | <code>EndDate</code>          | Employment end date             | Leave blank if still active |
  | <code>Title</code>            | Job title                       |                             |
  | <code>JobFunction</code>      | Job function or category        |                             |
  | <code>CreatedDate</code>      | Date the HR record was created  | Date auto-detected          |
  | <code>SourceTag</code>        | Label for this data source      |                             |

  ***

  **department** - Departments & org units

  | Field                  | Description                 | Notes |
  | ---------------------- | --------------------------- | ----- |
  | <code>Name</code>      | Department or org unit name |       |
  | <code>SourceTag</code> | Label for this data source  |       |

  ***

  **activity\_analysis** - Audit logs & events

  | Field                                | Description                       | Notes                                                    |
  | ------------------------------------ | --------------------------------- | -------------------------------------------------------- |
  | <code>ActivityType</code>            | OBS activity category             | Use <code>map\_value</code> to convert raw event names   |
  | <code>Timestamp</code>               | When the event occurred           | Date auto-detected                                       |
  | <code>ActorAccountId</code>          | Account that performed the action | Use <code>prefixed\_id</code> transform                  |
  | <code>IpAddress</code>               | IP address the action came from   |                                                          |
  | <code>Activity</code>                | Human-readable description        |                                                          |
  | <code>ApplicationActivityType</code> | Raw event name from your system   |                                                          |
  | <code>AffectedObjectType</code>      | Type of object affected           | e.g. <code>Account</code>, <code>ResourceInstance</code> |
  | <code>AffectedObjectId</code>        | ID of the affected object         |                                                          |
  | <code>ErrorCode</code>               | Error code if the action failed   |                                                          |
  | <code>SourceTag</code>               | Label for this data source        |                                                          |

  ***

  **application\_instance** - IdP-managed applications

  | Field                   | Description                     | Notes |
  | ----------------------- | ------------------------------- | ----- |
  | <code>Name</code>       | Display name of the application |       |
  | <code>VendorName</code> | Name of the vendor or provider  |       |
  | <code>SourceTag</code>  | Label for this data source      |       |
</Accordion>

#### Minimum viable example

The simplest possible config - application users and roles, with a membership relationship:

<div style={{maxHeight: '280px', overflowY: 'auto', overflowX: 'auto'}}>
  ```yaml theme={null}
  source:
    s3:
      # Use "startedat=" if your rawdata/ has date-stamped subfolders.
      # Use "" (empty string) if your CSV folders are directly inside rawdata/.
      partition_prefix: "startedat="
      # rbac_sync_interval_hours: 3    # optional, default: 3 hours
      # activity_sync_interval_hours: 1 # optional, default: 1 hour

  data_files:
    users:
      subfolder: "users/"
    roles:
      subfolder: "roles/"
    user_roles:
      subfolder: "user_roles/"

  objects:
    # Entity: Application user
    - obs_object_name: "applicationaccount"
      data_file: "users"
      id:
        type: "prefixed"
        prefix: "user"
        source_column: "email"        # CSV column used as the unique identifier
      field_mappings:
        - field: "Name"
          column: "full_name"
        - field: "Email"
          column: "email"
        - field: "Alias"
          column: "username"
        - field: "Enabled"
          column: "status"
          transform: "map_value"      # convert status text to true/false
          value_map:
            active: "true"
            inactive: "false"
          default_value: "false"
        - field: "CreatedDate"
          column: "created_at"        # date format is auto-detected

    # Entity: Role
    - obs_object_name: "role"
      data_file: "roles"
      id:
        type: "prefixed"
        prefix: "role"
        source_column: "role_id"
      field_mappings:
        - field: "Name"
          column: "role_name"
        - field: "Description"
          column: "description"

    # Relationship: User → Role membership
    - obs_object_name: "applicationaccount_memberof_role"
      data_file: "user_roles"
      links:
        - field: "ApplicationAccountId"
          prefix: "user"              # must match the prefix used in applicationaccount above
          column: "user_email"
        - field: "RoleId"
          prefix: "role"             # must match the prefix used in role above
          column: "role_id"
  ```
</div>

#### Full example (HR system)

This example models an HR export with employees, departments, and org hierarchy:

<div style={{maxHeight: '280px', overflowY: 'auto', overflowX: 'auto'}}>
  ```yaml theme={null}
  source:
    s3:
      partition_prefix: "startedat="

      # Optional: override the default sync schedules
      rbac_sync_interval_hours: 3
      activity_sync_interval_hours: 1

  data_files:
    employees:
      subfolder: "employees/"
    department_hierarchy:
      subfolder: "department_hierarchy/"

  objects:
    # Maps each employee row to a Person in Oleria
    - obs_object_name: "person"
      data_file: "employees"
      id:
        type: "prefixed"
        prefix: "person"
        source_column: "Email"
      field_mappings:
        - field: "Name"
          column: "Name"
        - field: "PrimaryPersonalEmail"
          column: "Email"
        - field: "CreatedDate"
          column: "Creation date"

    # Maps each employee row to an Employee record
    - obs_object_name: "employee"
      data_file: "employees"
      filters:
        - column: "Employee ID"
          op: "not_empty"             # skip rows where Employee ID is blank
      id:
        type: "prefixed"
        prefix: "emp"
        source_column: "Email"
      field_mappings:
        - field: "WorkName"
          column: "Preferred full name"
        - field: "PrimaryWorkEmail"
          column: "Email"
        - field: "EmployeeNumber"
          column: "Employee ID"
        - field: "StartDate"
          column: "Start date"
        - field: "Title"
          column: "Title"

    # Maps each employee row to a Department
    - obs_object_name: "department"
      data_file: "employees"
      id:
        type: "prefixed"
        prefix: "dept"
        source_column: "Department"
      field_mappings:
        - field: "Name"
          column: "Department"

    # Relationship: Person → Employee
    - obs_object_name: "person_is_employee"
      data_file: "employees"
      links:
        - field: "PersonId"
          prefix: "person"
          column: "Email"
        - field: "EmployeeId"
          prefix: "emp"
          column: "Email"

    # Relationship: Manager → Employee
    - obs_object_name: "employee_manages_employee"
      data_file: "employees"
      links:
        - field: "ManagerEmployeeId"
          prefix: "emp"
          column: "Manager Email"
        - field: "EmployeeId"
          prefix: "emp"
          column: "Email"

    # Relationship: Employee → Department
    - obs_object_name: "employee_serving_in_department"
      data_file: "employees"
      links:
        - field: "EmployeeId"
          prefix: "emp"
          column: "Email"
        - field: "DepartmentId"
          prefix: "dept"
          column: "Department"

    # Relationship: Department hierarchy (parent → child)
    - obs_object_name: "department_contains_department"
      data_file: "department_hierarchy"
      links:
        - field: "ParentDepartmentId"
          prefix: "dept"
          column: "ParentDepartment"
        - field: "ChildDepartmentId"
          prefix: "dept"
          column: "ChildDepartment"
  ```
</div>

**Complete sample data** - each includes a ready-to-use `config.yaml` and matching CSV files:

* [HR System Sample](https://github.com/roanokedatasecurity/oleria-docs/tree/main/public/samples/custom-application/sample-hr-mapping)
* [PostgreSQL Sample](https://github.com/roanokedatasecurity/oleria-docs/tree/main/public/samples/custom-application/sample-postgres-mapping)

## Grant Oleria Access to Your S3 Bucket

Oleria reads your data using a dedicated AWS IAM role. You need to add a bucket policy that allows this role to read your files.

<Warning>
  Do not make your S3 bucket public. Oleria accesses your bucket securely using a private IAM role - no public access is required or recommended. Oleria only needs read-only access and will never write to, modify, or delete any files in your bucket. If this policy is not in place, your integration will appear connected in the UI but no data will be synced.
</Warning>

<Steps>
  <Step title="Get the Oleria IAM role ARN">
    Your Oleria Customer Success team will provide you with the exact IAM role ARN for your environment. The role follows this format:

    ```
    arn:aws:iam::<aws_account_id>:role/<tenant_name>-api-ecs-task-execution-role
    ```
  </Step>

  <Step title="Add a bucket policy">
    Go to your S3 bucket in the **AWS Console** → **Permissions** tab → **Bucket policy** → **Edit**, and paste the following policy. Replace the placeholders with your values and select **Save changes**.

    <div style={{maxHeight: '280px', overflowY: 'auto'}}>
      ```json theme={null}
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "AllowOleriaReadAccess",
            "Effect": "Allow",
            "Principal": {
              "AWS": "arn:aws:iam::<aws_account_id>:role/<tenant_name>-api-ecs-task-execution-role"
            },
            "Action": [
              "s3:GetObject",
              "s3:ListBucket"
            ],
            "Resource": [
              "arn:aws:s3:::your-bucket-name",
              "arn:aws:s3:::your-bucket-name/your-folder-prefix/*"
            ]
          }
        ]
      }
      ```
    </div>
  </Step>

  <Step title="Update the KMS key policy (if applicable)">
    If your S3 bucket is encrypted with a customer-managed KMS key, update the KMS key policy. Go to **AWS KMS** → select your key → **Key policy** → **Edit**, and add the following statement:

    <div style={{maxHeight: '240px', overflowY: 'auto'}}>
      ```json theme={null}
      {
        "Sid": "AllowOleriaDecrypt",
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::<aws_account_id>:role/<tenant_name>-api-ecs-task-execution-role"
        },
        "Action": [
          "kms:Decrypt",
          "kms:DescribeKey"
        ],
        "Resource": "*"
      }
      ```
    </div>

    <Warning>Without this KMS policy, Oleria will not be able to decrypt and read your files even if the bucket policy is correctly configured.</Warning>
  </Step>
</Steps>

## Connect the Integration in Oleria

<Steps>
  <Step title="Open the integration">
    Go to your Oleria workspace, select **Integrations** → select **Custom Application**.

    <img src="https://mintcdn.com/oleria/9eNE4qr_2dvBWbfD/images/integrations/custom-application/step-1.png?fit=max&auto=format&n=9eNE4qr_2dvBWbfD&q=85&s=81ba3bf13a11885a7723a43e7c69d91d" alt="Integrations page showing the Custom Application option" width="3438" height="1608" data-path="images/integrations/custom-application/step-1.png" />
  </Step>

  <Step title="Complete the connection form">
    Fill in the connection form that appears:

    * **Application Name** - a label for this integration in Oleria. Use a name that describes your data source (for example: `Lattice HR`, `Databricks Prod`, `Internal App`).
    * **S3 Data Path** - the full S3 URL to the root folder of your data. This is the folder that contains your `config/` and `rawdata/` subfolders. Example: `s3://your-bucket-name/your-prefix/`

          <img src="https://mintcdn.com/oleria/9eNE4qr_2dvBWbfD/images/integrations/custom-application/step-3.png?fit=max&auto=format&n=9eNE4qr_2dvBWbfD&q=85&s=fd63d1dee66dba713702acdd9d80f842" alt="Connection form with Application Name and S3 Data Path fields" width="1268" height="1804" data-path="images/integrations/custom-application/step-3.png" />
  </Step>

  <Step title="Authenticate">
    Select **Authenticate**. Oleria will verify it can reach your S3 bucket and read your configuration file.
  </Step>

  <Step title="Confirm the connection">
    Once connected, find your integration in the **Connected Integrations** section of the Integrations page. Oleria will sync based on the intervals defined in your `config.yaml` - by default, access data every **3 hours** and activity logs every **1 hour**. You can override these using `rbac_sync_interval_hours` and `activity_sync_interval_hours` in your config. Oleria will only process data when new files are available.

    <img src="https://mintcdn.com/oleria/9eNE4qr_2dvBWbfD/images/integrations/custom-application/step-4.png?fit=max&auto=format&n=9eNE4qr_2dvBWbfD&q=85&s=d8942217560d1a4a4a523a000c6ef8d2" alt="Connected Integrations section showing Custom Application as active" width="414" height="376" data-path="images/integrations/custom-application/step-4.png" />
  </Step>
</Steps>

## Contact us

For questions about this integration, contact us at [support@oleria.com](mailto:support@oleria.com).