Summary

Synopsis

Discussion

Initialization Code

At the beginning of your script, add the following code:

import com.palamida.appcore.properties.ApplicationProperties
import com.palamida.script.*
def coreServer = ApplicationProperties.getInstance().getCoreServer()

This sets the coreServer variable to instantiate the covers in the com.palamida.script namespace.

AdminServiceCover

The AdminServiceCover handles actions related to user, team, project and policy management. It also manages metadata values related to users, teams, policies and tasks, and can perform some utility functions, such as getting server configuration.

Instantiate the service cover:

def adminSrv = new AdminServiceCover(coreServer)

Get a user from the database:

def user = adminSrv.getUser("martha")
// print full name
println([user.getFirstName(), user.getLastName()].join(" "))

Create a user list:

NOTE: The API supports creating user lists of the Mixed type only. For other list types (Requester, Reviewer, Participant), create the user list in the web UI. After that, you can update the users in the list through the API.

def listName = "my user list"
def listOwner = "martha"
def description = "User list created from API"
def userNames = ["alice", "bob", "carol"] as Set
adminSrv.createOrUpdateUserList(listName, description, listOwner, userNames, null)

Get user names and email addresses from a user list:

def userNames = adminSrv.getUserLoginsForUserList(listName)
def userEmails = userNames.collect { adminSrv.getUser(it).getEmail() }

Create a user. User data is represented by the UserDataModel class:

import com.palamida.appsec.BusinessException
import com.palamida.script.model.UserDataModel
def login = "alice"
try {
    adminSrv.getUser(login)
    println "User already exists"
} catch(BusinessException e) {
    // set user attributes
    def password = "password"
    def firstName = "Alice"
    def middleName = "Louise"
    def lastName = "Thompson"
    def email = "alice@corp.com"
    def jobTitle = "Job Title"
    def telephone = "415-777-9400"
    def fax = "415-777-5800"
    def location = "San Francisco"
    def businessUnit = "Business Unit"
    def question = "security question"
    def answer = "the answer"

    // create the user
    def userData = new UserDataModel(password, firstName, middleName, lastName, email, jobTitle, telephone,
                        fax, location, businessUnit, question, answer, 0)
    def userId = adminSrv.createOrUpdateUser(login, userData)
}

Assign the role of Requester to a user:

def login = "alice"
def user = adminSrv.getUser(login)
adminSrv.setUserRoleAsRequester(user.getId(), true)

Get defined teams:

def teams = adminSrv.getTeamNames()
if (teams.isEmpty())
    println "No teams defined"

Create a team:

def teamName = "product-team"
def description = "Team created from API"
assert !adminSrv.getTeamNames().contains(teamName), "Team already exists"
def teamId = adminSrv.createOrUpdateTeam(teamName, description)

Create a project:

def projectName = "Product 3.0"
def description = "Project created from API"
def owner = "martha"
assert !adminSrv.getProjectNames(teamName).contains(projectName), "Project already exists"
def projectId = adminSrv.createOrUpdateProject(projectName, description, teamName, owner)

Get a project owner's email address:

def userData = adminSrv.getOwnerForProject(teamName, projectName)
println user.getEmail()

Scan all workspaces in a project. For scanning individual workspaces, or running workspace reports, see scheduling scans:

adminSrv.scanProject(teamName, projectName)
Get a project id:

def teamName = "product-team"
def projectName = "Product 3.0"
def projectId = adminSrv.getProjectId(teamName, projectName)
Get a workspace id:

def workspaceName = "product_3_0_src"
def workspaceId = adminSrv.getWorkspaceUuid(teamName, projectName, workspaceName)

To create workspaces, see the ProjectDataCover.

ProjectDataCover

The ProjectDataCover handles project configuration, user roles, requests, and workspace management.

Instantiate the cover:

def projSrv = new ProjectDataCover(coreServer)
// use admin service for getting ids
def adminSrv = new AdminServiceCover(coreServer)

Get a project id:

def teamName = "product-team"
def projectName = "Product 3.0"
def projectId = adminSrv.getProjectId(teamName, projectName)

Set the request form for a project:

def formName = "Short Request Form Definition"
projSrv.setRequestForm(projectId, formName)

Set a review level called Legal and assign reviewers for a project:

def reviewLevel = "Legal"
def userListName = "legal reviewers" // required. only users with the Reviewer role will be added.
def userNames = ["martha"] // can be empty
projSrv.updateProjectReviewers(projectId, reviewLevel, userNames, userListName)

Assign the role of Requester for a project to some user lists:

def userListNames = ["requesters"] // all lists must be of the Requester type.
projSrv.setProjectRequestersUserLists(projectId, userListNames)

Assign the role of Auditor for a project to some user lists:

def userListNames = ["auditors"] // all lists must be of the Participant type.
projSrv.setProjectAuditorsUserLists(projectId, userListNames)

Set project status to In Progress:

projSrv.setProjectStatusToPending(teamName, projectName)

Create a workspace. Read more on configuring workspaces:

def workspaceName = "product_3_0_src"
def scannerAlias = "Scanner1"
def success = projSrv.createWorkspace(teamName, projectName, workspaceName, scannerAlias)
assert success, "Unable to create workspace"
println "Workspace created"

Delete a workspace:

projSrv.deleteWorkspace(projectName, scannerAlias, workspaceName)

Create a request for a project, and submit it:

// requester and project
def requesterUsername = "alice"
def projectId = adminSrv.getProjectId(teamName, projectName)

// component/version/license. see reference data service for getting ids
def componentId = 64223  // zlib
def versionId = 79532    // "1.2.3"
def licenseId = 8        // "zlib/libpng license"

// request attributes. the available attributes and values are defined by the request form
def attributes = [
    "Encryption": "No",
    "Modifications": "No",
    "ComponentUsage": "Redistribution",
    "Redistribution": ["Embedded", "API"].join("\n")  // separate multiple options with \n
]

// comment to add (max 5,000 characters)
def comment = "Request created through API"

// create request as draft
def status = "draft"
def requesterId = adminSrv.getUser(requesterUsername).getId()
def requestId = projSrv.createRequest(
    projectId, requesterId, status, componentId, versionId, licenseId, comment, attributes
)

// submit request
projSrv.submitDraftRequest(requestId)

Get inventory information for a project:

def inventory = projSrv.getInventoryForProject(teamName, projectName)
inventory.each { inv ->
    println "Name: ${inv.getAuditGroupLabel()}"
    println "Component: ${inv.getComponentName()} ${inv.getVersionModel()?.getComponentVersionName()}"
    println "License: ${inv.getLicense()?.getLicenseName()} \n"
}

Create a checklist item for an inventory item and assign it to a user:

def inventoryId = 1
def message = 'Please review this item for legal issues'
// The type codes for checklist items are:
// 1 - General, 2 - Legal, 3 - Remediation, 4 - Security
def itemType = 2
def assignee = 'martha' // user login, must be a project participant
def itemId = projSrv.createChecklistItem(inventoryId, itemType, message)
if (itemId)
    projSrv.reassignInventoryChecklistItem(itemId, assignee)

Get all the requests for a project.

def requests = projSrv.getAllRequestsForProject(teamName, projectName)
requests.each { request ->
    println "ID: ${request.getId()}"
    println "Component: ${request.getComponentName()} ${request.getComponentVersion()}"
    println "License: ${request.getLicenseName()} \n"
}

AuditorServiceCover

The AuditorServiceCover performs functions similar to the Detector client interface. It manages tags and groups, using file path information from workspaces.

Instantiate the service cover:

def auditSrv = new AuditorServiceCover(coreServer)
// use admin service for getting project/workspace ids
def adminSrv = new AdminServiceCover(coreServer)

Print all the groups for a project:

def teamName = "product-team"
def projectName = "Product 3.0"
def projectId = adminSrv.getProjectId(teamName, projectName)
assert projectId, "Project does not exist"
auditSrv.getGroupIds(projectId).each {
    groupId ->
    def groupDetails = auditSrv.getGroupDetails(groupId)
    println groupDetails.getName()
}

Get files paths in a workspace with the tag "Contains Indicators" = "Yes":

def workspaceName = "product_3_0_src"
def workspaceId = adminSrv.getWorkspaceUuid(team, projectName, workspaceName)
def tagName = "Contains Indicators"
def tagValue = "Yes"
def filePaths = auditSrv.getFilesWithTag(tagName, tagValue, workspaceId)

Get files containing license matches:

def filePaths = auditSrv.getFilesWithTag("Contains License Matches", "Yes", workspaceId)

Get files containing email/URL matches:

def filePaths = auditSrv.getFilesWithTag("Contains Email/URL Matches", "Yes", workspaceId)

Get files containing source code fingerprints:

def filePaths = auditSrv.getFilesWithTag("Contains Source Matches", "Yes", workspaceId)

Tag files in a workspace:

auditSrv.applyTagToFiles(filePaths, tagName, tagValue, workspaceId)

Create a new tag:

def tagName = "my tag"
def description = "Tag created through API"
assert !auditSrv.getTag(tagName), "Tag already exists"
auditSrv.createTag(tagName, description, true)

ReferenceDataServiceCover

The ReferenceDataServiceCover accesses component, version, license and vulnerability information.

Instantiate the service cover:

def refDataSrv = new ReferenceDataServiceCover(coreServer)

Find components by name:

def componentName = "zlib"
def componentIds = refDataSrv.getComponents(componentName)
assert componentIds.contains(64223)

Get component information for zlib (id 64223). Refer to the ComponentDataModel javadoc:

def componentId = 64223
def component = refDataSrv.getComponent(componentId)
assert component.getName().equals("zlib")
assert component.getUrl().equals("http://www.gzip.org/zlib/")

Get associated licenses from component. Refer to the LicenseDataModel javadoc:

def licenseIds = refDataSrv.getLicensesForComponent(componentId)
def licenses = licenseIds.collect { refDataSrv.getLicense(it) }
def licenseNames = licenses.collect { it.getName() }
assert licenseNames.contains("zlib/libpng License")

Get number of vulnerabilities for libtiff 3.7.3:

def componentName = "libtiff"
def versionName = "3.7.3"
def componentId = refDataSrv.getComponents("libtiff")[0]
def versionId = refDataSrv.getComponentVersion(componentId, versionName)
println refDataSrv.getNoOfVulnerabilities(versionId)

Get the license obligations for a license:

def licenseId = 21
def license = refDataSrv.getLicenseNotDataModel(licenseId)
def obligationIds = license.getObligations().collect { it.getId() }
obligationIds.each { obId ->
    def obligation = refDataSrv.getLicenseObligationByID(obId)
    println "Required: ${obligation.isRequired()}"
    println "Type: ${obligation.getType()}"
    println "Organization: ${obligation.getOrganization()}"
    println "Priority: ${obligation.getPriority()}"
    println "Trigger Action: ${obligation.getTriggerAction()}"
    println "Instruction: ${obligation.getInstruction()}"
    println "License Text Section: ${obligation.getLicenseTextSection()}"
    println "License Text Fragment: ${obligation.getLicenseTextFragment()}"
}

MetadataServiceCover

The MetadataServiceCover class manages the creation of metadata fields. Metadata fields can be defined for various entities (e.g. project, component), and can be of various types (e.g. boolean, text).

Instantiate the service cover:

def mdSrv = new MetadataServiceCover(coreServer)

Get the id for the entity project:

// available: team, project, group, component, component_version,
//            license, request, inventory, policy
def entityId = mdSrv.getEntityId("project")

Create a metadata text field called some_project_info for the entity project:

import com.palamida.appsec.model.MDDefinition
def fieldName = "some_project_info"
def displayName = "Some Project Info"
def fieldType = MDDefinition.MDDatatype.Text
def field = new MDDefinition(fieldType, entityId, fieldName, displayName)
def fieldId = mdSrv.addMDDefinition(field)

Store a value for a metadata field for a particular project:

def teamName = "product-team"
def projectName = "Product 3.0"
def fieldName = "some_project_info"
def value = "some value"
def adminSrv = new AdminServiceCover(coreServer)
def entityId = mdSrv.getEntityId("project")
def projectId = adminSrv.getProjectId(teamName, projectName) as Long
def fieldId = mdSrv.getMetadataDefinitionId(entityId, fieldName)
mdSrv.addOrUpdateMetadataValue(fieldId, projectId, value)

Read a value from a metadata field for a particular project:

def values = mdSrv.getValue(fieldId, (long) projectId)
if (values) {
    def value = values[0]
    println "Value: $value"
}

WorkspaceCover

The WorkspaceCover class provides access to workspace configuration, and serves as the entry point for accessing scan results.

Identifying workspaces

Each workspace resides on a FNCI Scan server. Scan servers are defined in the core.properties file on the Core server.

scan.server.aliases=Scanner1,Scanner2
scan.server.Scanner1.web=http://scan1.corp.com:8888/palamidaScanEngine
scan.server.Scanner1.rmi=rmi://scan1.corp.com:1099
scan.server.Scanner2.web=http://scan2.corp.com:8888/palamidaScanEngine
scan.server.Scanner2.rmi=rmi://scan2.corp.com:1099

Workspaces are opened using a URI of the form:

http(s)://<scanServerAddress>/<workspaceName>

For example, the URI for the workspace product_3_0-src on Scanner1 would be (without SSL):

http://scan1.corp.com/product_3_0-src

ScriptRunner binds a WorkspaceLocatorCover instance to the variable locator, which is used to schedule scans and open workspaces. To get a scan server uri from a scanner alias, you can use the following method:

def adminSrv = new AdminServiceCover(coreServer)
def scannerAlias = "Scanner1"
def props = adminSrv.getPropsMapFromCoreServerPropertiesFile("core.properties")
def key = "scan.server." + scannerAlias + ".web"
assert props.containsKey(key), "Scanner not found"
def url = props.get(key).toURL()
def scannerUri = url.getProtocol() + "://" + url.getHost()

Scheduling a scan

Get the SchedulerServiceCover instance for a scan server from the workspace locator:

def scannerUri = "http://scan1.corp.com:8888/palamidaScanEngine"
def schedSrv = locator.getSchedulerService(scannerUri)

Schedule a scan of a workspace. See also the SchedulableTask javadoc:

def workspaceName = "product_3_0-src"
def uri = [scannerUri, workspaceName].join("/")
def task = schedSrv.addScanTask(uri)
println "Task added with id ${task.getScheduleId()}"

Schedule a workspace report:

def reportName = "Third-Party Indicators Report"
def task = schedSrv.addReportTask(uri, reportName)

Schedule scan and reports:

def reportNames = [
    "Scanned Files Report",
    "Third-Party Indicators Report"
]
def tasks = schedSrv.addScanTask(uri, reportNames)

Get the active task (if any):

def task = schedSrv.getActiveTask()
if (task)
    println "Task with id ${task.getScheduleId()} is active"
else
    println "No active tasks for this scan server"

Get tasks in the queue:

def tasks = schedSrv.getTasksInQueue()

Cancel a task:

schedSrv.cancelTask(task)

Get the status of a task. For possible statuses, see the TaskState javadoc:

import com.palamida.scheduler.TaskState
def state = task.getState()
if (state.equals(TaskState.COMPLETED))
    println "Task completed successfully"

Opening a workspace

Opening a workspace connects to a Scan server to access the scanned codebase and the database containing the scan results.

You can pass a workspace URI to ScriptRunner with the -w option, and it will handle opening and closing the workspace, binding the workspace variable to the WorkspaceCover instance in scripts.

To manually open and close workspaces:

def uri = "http://scan1.corp.com/product_3_0-src"
def workspace
try {
    workspace = locator.openWorkspace(uri)
    // ...

} finally {
    if (workspace)
        workspace.close()
    locator.destroy()
}

Configuring a workspace

Set the workspace description:

workspace.setDescription("Workspace configured through API")

Get the scan paths:

def fileset = workspace.getScanFileSet()
def filePaths = fileset.getPaths()

Set the path to scan:

def filePaths = ["/data/product-team/product/3.0/src"] as Set
def fileset = locator.createFileset()
fileset.setPaths(filePaths)
workspace.updateScanFileSet(fileset)

Add 'temp' directories to exclusions:

Set exclusions = []
def fileset = workspace.getScanFileSet()
exclusions.addAll(fileset.getExclusions())
exclusions.add('**\\temp')
fileset.setExclusions(exclusions)
workspace.updateScanFileSet(fileset)

Add the search term 'affero' to the workspace. See the SearchString class:

import com.palamida.workspace.SearchString
def searchString = new SearchString()
searchString.setValue("affero")
searchString.setSource(SearchString.USER_SOURCE)
searchString.setEnabled(true)
workspace.getInventoryService().addSearchString(searchString)

Remove the search term 'generated' from the workspace, if it exists.

def searchStrings = workspace.getInventoryService().getSearchStrings()
def searchString = searchStrings.find { it.getValue().equals("generated") }
if (searchString)
    workspace.getInventoryService().removeSearchString(searchString)

Disable Source Code Fingerprint scanning for the workspace:

workspace.setUseSourceCodeFingerprintScanning(false)

Set files with the extension '.src' to be scanned as JavaScript for source code fingerprinting:

def extension = "src"
// check workspace.getScfTypes() for available language types
def langType = "javascript"
def extensions = workspace.getScfExtensions(langType) as List
extensions.add(extension)
workspace.setScfExtensions(langType, extensions)

ScanResultServiceCover

The ScanResultServiceCover queries a workspace's scan results for details about indicator types.

The scan result service is instantiated from an open workspace cover (see above):

def scanResultSrv = workspace.getScanResultService()
// use auditor service to get node, path & tag information
def auditSrv = new AuditorServiceCover(coreServer)

Get the names all detected copyright holders:

def copyrightHolders = scanResultSrv.getCopyrightOwners().collect { it.getName() }

Get a map of copyright holders to file paths:

def copyrightFiles = [:]
scanResultSrv.getCopyrightOwners().each {
    owner ->
    def copyrightMatches = scanResultSrv.getCopyrightMatchesForCopyrightOwner(owner)
    def nodeIds = copyrightMatches.collect { it.getNodeId() }
    def fileIds = auditSrv.getFileIdsForNodeIds(nodeIds)
    def filePaths = auditSrv.getFilePathsForIDs(fileIds)
    copyrightFiles.put(owner.getName(), filePaths)
}

Get a map of detected licenses to file paths:

def workspaceId = workspace.getUuid()
def licenseFiles = [:]

// use reference data service to lookup license names
def refDataSrv = new ReferenceDataServiceCover(coreServer)
def licenseCache = [:]  // cache lookup table

auditSrv.getFilesWithTag("Contains License Matches", "Yes", workspaceId).each {
    filePath ->
    def nodeId = auditSrv.getNodeIds([filePath]).iterator().next()
    scanResultSrv.getLicenseMatchesForNodeId(nodeId).each {
        def licenseId = it.getLicenseId()
        if (!licenseCache.containsKey(licenseId))
            licenseCache[licenseId] = refDataSrv.getLicense(licenseId).getName()
        def licenseName = licenseCache[licenseId]
        if (!licenseFiles.containsKey(licenseName))
            licenseFiles[licenseName] = [] as Set
        licenseFiles[licenseName].add(filePath)
    }
}