Skip to Content

The powerful capabilities which SAP’s Cloud ERP SAP Business ByDesign provides for GDPR have been enhanced with 1802 (here you can see what has been delivered before 1802.). With ByDesign 1802 your data privacy officers will be enabled to:

  • Block the deletion of natural persons e.g. because of a pending lawsuit
  • Manage read access logs for special categories of personal data like religion, bank etc.
  • Find leads without reference to account master data and delete it
  • Find and delete data for contacts of partners or accounts

 

The demo video below gives you an impression on how this works in our 1802 release:


Block for deletion

Data privacy experts are able to disallow the deletion of natural persons data e.g. due to a legal hold or other reasons. You can:

  1. Search and mark natural persons like private account or employees as blocked for removal.
  2. Add reason and description for the removal block.


Read Access Logging

Data privacy experts can configure which predefined business partner attributes are to be treated as special category of data and activate read access logging for sensitive data of natural persons.

  1. Activate or deactivate predefined field groups for read access logging including change log.
  2. Extension fields can be marked as special category either by key user or PDI. Field group configuration can be extended by PDI.
  3. PDI allows to mark partner created BO fields as special.
  4. Download daily read access logs by webservice or UI within 14 days. Log is provided as XML files to be archived with external applications.

The read access logs will monitor access not with the normal user interface separated for transactional, query or analytical requests but also for technical interfaces like webservices or OData calls.

As this might depending on the number of fields and users you want to monitor create huge data volumes the logs will be stored in ByDesign only for a limited timeframe. After this they will be deleted. Therefore we have implemented a webservice which allows to automatically retrieve the ZIP-files and to store them securly on a local device or in a secure archivieng solution which might import the raw XML data in order to make it available to e.g. auditors.

Here you can find a sample Python script to consume the read access log webservice. Be aware that this is just an example for which we exclude any liability and which is not intended for productive use.


Leads without accounts

Data privacy experts are able to search and delete Leads which store personal data without reference to an account. To do so you have to:

  1. Search all Leads which store address & contact data without referencing an account in the newly introduced leads view in the data privacy work center
  2. Trigger the deletion or disclose the data to the natural person

Be aware that leads without reference to accounts will not be found in the data removal or disclose view as they store the personal data in the lead itself.

 


Disclaimer

The information provided in this blog should not be considered as legal advice or replace legal counsel for your specific needs. Readers are cautioned not to place undue reliance on these statements and they should not be relied upon in making purchasing decisions or for achieving compliancy to legal regulations.


Sample Python Script to consume the read access log web service

Please be aware that this is just an example to illustrate how to use the webservice. It is not intended for productive usage and we exclude any liability.

Here you can find a sample Python script to consume the read access log webservice.

 

README

# ByDDownloadReadAccessLog
The ByDesign read access log will be deleted after two weeks. In the meantime it needs to be downloaded
to a local file system. ByDesign offers a web service to automate this process.
 
This project provides an example implementation on how automatic download can be accomplished.

## Run
Run using "download.py". Runs with vanilla Python 3 (no special libraries needed).

## Prerequists
Install Python 3 (we used version 3.6)

Create a configuration file `settings.ini`. Use `example_settings.ini` as starting point.

When running from company internal systems it's likely you need to 
specify an http proxy. You do this by specifying a value for `proxy` in `settings.ini` or
by setting the environment variable `HTTPS_PROXY`. 

 

import sys, os.path, configparser, json, base64, xml.etree.ElementTree
import urllib.request, urllib.error, urllib.parse
import collections
import pprint

#
# Use the Web Service QueryReadAccessLogIn to download new read access log (RAL) files
#
# Read Access Log (RAL) represent ZIP files. They contain UUIDs, start and end timestamps
# and (if requested) file content containing the read access log of the corresponding time frame.
#
# Works two steps:
#  1. Download a list of all new RAL files /omitting/ the file content. By ommitting the file
#     content, the response size will be quite small
#  2. Download the content of the above RAL files. To avoid long service calls in case of large
#     file content, the files are downloaded one by one.
#

#
# Web Service Request bodies
#

# Request to get a list of all RAL files in the system.
# Set TransmissionRequestCode to "2" to avoid reading the actual file content
LIST_ALL_RAL_FILES_REQUEST = """<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:glob="http://sap.com/xi/SAPGlobal20/Global">
   <soapenv:Header/>
   <soapenv:Body>
      <glob:ReadAccessLogByElementsQuery_sync>
         <ProcessingConditions>
            <QueryHitsUnlimitedIndicator>true</QueryHitsUnlimitedIndicator>
         </ProcessingConditions>
         <RequestedElements ReadAccessLogTransmissionRequestCode="2" />
      </glob:ReadAccessLogByElementsQuery_sync>
   </soapenv:Body>
</soapenv:Envelope>"""

# Request to get a list of RAL files after a certain point in time (but *not* the file content!)
#
# Set TransmissionRequestCode to "2" to avoid reading the actual file content
# Set IntervalBoundaryTypeCode to 9 to specify we want files /greater or equal/ than the
# given timestamp
LIST_NEW_RAL_FILES_REQUEST = """<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:glob="http://sap.com/xi/SAPGlobal20/Global">
   <soapenv:Header/>
   <soapenv:Body>
      <glob:ReadAccessLogByElementsQuery_sync>
         <ReadAccessLogSelectionByElements>
            <SelectionByStartDateTime>
               <InclusionExclusionCode>I</InclusionExclusionCode>
               <IntervalBoundaryTypeCode>9</IntervalBoundaryTypeCode>
               <LowerBoundaryDateTime>%s</LowerBoundaryDateTime>
            </SelectionByStartDateTime>
         </ReadAccessLogSelectionByElements>
         <ProcessingConditions>
            <QueryHitsUnlimitedIndicator>true</QueryHitsUnlimitedIndicator>
         </ProcessingConditions>
         <RequestedElements ReadAccessLogTransmissionRequestCode="2" />
      </glob:ReadAccessLogByElementsQuery_sync>
   </soapenv:Body>
</soapenv:Envelope>"""

# Request to get the file content for a specific RAL file specified by it's UUID
GET_SPECIFIC_FILE_CONTENT_REQUEST = """<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:glob="http://sap.com/xi/SAPGlobal20/Global">
   <soapenv:Header/>
   <soapenv:Body>
      <glob:ReadAccessLogByElementsQuery_sync>
         <ReadAccessLogSelectionByElements>
            <SelectionByUUID>
               <InclusionExclusionCode>I</InclusionExclusionCode>
               <IntervalBoundaryTypeCode>1</IntervalBoundaryTypeCode>
               <LowerBoundaryUUID>%s</LowerBoundaryUUID>
            </SelectionByUUID>
         </ReadAccessLogSelectionByElements>
      </glob:ReadAccessLogByElementsQuery_sync>
   </soapenv:Body>
</soapenv:Envelope>"""

#
# Web service execution
#

# Named tuples to be returned by web service calls:
ListedRalFile  = collections.namedtuple("ListedRalFile", ["uuid", "start_date_time", "end_date_time", "log_size"])
RalFileContent = collections.namedtuple("RalFileContent", ["content", "filename"])


class WebServiceClient:
    def __init__(self, byd_host_name: str, username: str, password: str, proxy: str = None):
        """
        Establish a connection with the corresponding host
        """
        self._web_service_url = "https://%s/sap/bc/srt/scs/sap/queryreadaccesslogin" % byd_host_name

        # Establish handlers for authentication and proxy handling
        handlers = [self._get_basic_auth_handler(username, password)]
        if proxy is not None:
            handlers.append(self._get_proxy_handler(proxy))
        urllib.request.install_opener(urllib.request.build_opener(*handlers))

    def _get_proxy_handler(self, proxy:str) -> urllib.request.BaseHandler:
        """
        Get a handler object for proxy handling
        """
        return urllib.request.ProxyHandler({'https':proxy})

    def _get_basic_auth_handler(self, username: str, password: str) -> urllib.request.BaseHandler:
        """
        Get a handler object for basic authentication handling
        """
        # Sub-URLs of "url" will later be opened with given username and password
        pw_manager = urllib.request.HTTPPasswordMgrWithDefaultRealm()
        pw_manager.add_password(None, self._web_service_url, username, password)
        return urllib.request.HTTPBasicAuthHandler(pw_manager)

    def _execute_web_service(self, request_body: str) -> str:
        """
        Execute web service with given body returning the response body
        """
        request = urllib.request.Request(self._web_service_url, request_body.encode())
        request.add_header("Content-Type", "text/xml")
        return urllib.request.urlopen(request).read()

    def _extract_ral_file_list(self, response_body: str) -> [ListedRalFile]:
        """
        From the response body (an XML string) extract all log RAL files
        """
        response_xml = xml.etree.ElementTree.fromstring(response_body)
        ral_files = [
            ListedRalFile(
                uuid=xml_entry.find("UUID").text,
                start_date_time=xml_entry.find("StartDateTime").text,
                end_date_time=xml_entry.find("EndDateTime").text,
                log_size=xml_entry.find("LogSize").text
            ) for xml_entry in response_xml.iter("ReadAccessLog")]
        return ral_files

    def _extract_single_content(self, response_body: str) -> RalFileContent:
        """
        From the response body (an XML string) extract the content of the first RAL file.
        (It should be used when only the content of a single file is requested.)
        Return the content as named tuple with fields "content" and "filename"
        """
        response_xml = xml.etree.ElementTree.fromstring(response_body)
        for xml_entry in response_xml.iter("ReadAccessLog"):
            log_content_xml = xml_entry.find("LogData/LogContent")
            log_content_text = log_content_xml.text
            log_content_filename = log_content_xml.attrib["fileName"]
            return RalFileContent(content  = base64.b64decode(log_content_text),
                                  filename = log_content_filename)

    def list_new_ral_files(self, from_date: str) -> [ListedRalFile]:
        """
        Call the web service to retrieve list of RAL files starting with from_date or later.
        Returns a list of named tuples with fields
            uuid, start_date_time, end_date_time, log_size
        """
        response_body = self._execute_web_service(LIST_NEW_RAL_FILES_REQUEST % from_date)
        return self._extract_ral_file_list(response_body)

    def list_all_ral_files(self) -> [ListedRalFile]:
        """
        Call the web service to retrieve a list of all RAL files.
        Returns a list of named tuples with fields
            uuid, start_date_time, end_date_time, log_size
        """
        response_body = self._execute_web_service(LIST_ALL_RAL_FILES_REQUEST)
        return self._extract_ral_file_list(response_body)

    def get_specific_file_content(self, uuid: str) -> RalFileContent:
        """
        Call the web service to retrieve the content of a single log file specified by it's UUID.
        Return the content as named tuple with fields "content" and "filename".
        """
        response_body = self._execute_web_service(GET_SPECIFIC_FILE_CONTENT_REQUEST % uuid)
        return self._extract_single_content(response_body)


#
# Read configuration from file settings.ini
#

Configuration = collections.namedtuple("Config", ["hostname", "proxy", "user", "password", "target_dir"])

CONFIG_FILENAME = 'settings.ini'
CONFIG_BYD_TENANT = 'BYD_TENANT'
CONFIG_HOSTNAME = 'hostname'
CONFIG_USER = 'user'
CONFIG_PASSWORD = 'password'
CONFIG_PROXY = 'proxy'

def read_configuration() -> Configuration:
    """
    Read configuration from "settings.ini". This file must exist. It contains connection information as well as the
    target directory. See "example_settings.ini" for an example configuration.
    """
    try:
        parser = configparser.ConfigParser()
        parser.sections()
        parser.read(CONFIG_FILENAME)
        if CONFIG_PROXY in parser[CONFIG_BYD_TENANT]:
            proxy = parser[CONFIG_BYD_TENANT][CONFIG_PROXY]
        else:
            proxy = None
            
        config = Configuration(
            hostname  =parser[CONFIG_BYD_TENANT][CONFIG_HOSTNAME],
            user      =parser[CONFIG_BYD_TENANT][CONFIG_USER],
            password  =parser[CONFIG_BYD_TENANT][CONFIG_PASSWORD],
            proxy     =proxy,
            target_dir=parser['TARGET']['dirname'])
        if not os.path.isdir(config.target_dir):
            print(
                "The target directory '%s' specified in 'settings.ini' does not exist (or it is not a directory)" % config.target_dir)
            sys.exit(-1)
        return config

    except:
        print("You need a valid configuration file named 'settings.ini'. See 'example_settings.ini' for an example configuration.")
        sys.exit(-1)

#
# Main Program
#
if __name__ == "__main__":
    config = read_configuration()

    # Read end timestamp of last downloaded log file.
    try:
        with open(config.target_dir + "/" + "last_call.json", mode="r") as f:
            last_call_data = json.load(f)
            known_uuids_starting_with_max_time = frozenset(last_call_data["uuids_starting_with_max_time"])
            max_date_time                      = last_call_data["max_date_time"]

    except:
        # Set useful default values in case no reasonable last data could be retrieved
        known_uuids_starting_with_max_time = frozenset()
        max_date_time                      = None

    web_service_client = WebServiceClient(
        byd_host_name=config.hostname,
        username=config.user,
        password=config.password,
        proxy=config.proxy
    )

    #
    # First step: Download all new RAL files
    #
    try:
        if max_date_time:
            listed_ral_files = web_service_client.list_new_ral_files(max_date_time)
        else:
            listed_ral_files = web_service_client.list_all_ral_files()

    except urllib.request.URLError as err:
        print("URL error when talking to ByDesign Server:")
        print("... %s" % str(err))
        if 'HTTPS_PROXY' not in os.environ and not config.proxy:
            print("You might need to configure proxy settings. This can be done by setting the 'proxy' field in settings.ini.")
        sys.exit(-1)

    except urllib.request.HTTPError as err:
        print("Connection error when talking to ByDesign Server:")
        print("... %s" % str(err))
        sys.exit(-1)


    # Filter RAL files already downloaded
    new_ral_files = [ral_file for ral_file in listed_ral_files if ral_file.uuid not in known_uuids_starting_with_max_time]

    print("Found %d new read access log files" % len(new_ral_files))

    #
    # Second step: Retrieve content of all new log files one by one
    # and write them to target directory
    #
    count = 0
    for ral_file in new_ral_files:
        count += 1
        print("Treating RAL file %d of %d" % (count, len(new_ral_files)),
              flush=True, end="")
        file_content = web_service_client.get_specific_file_content(ral_file.uuid)
        with open(config.target_dir + "/" + file_content.filename, mode="wb") as f:
            f.write(file_content.content)
        print(" ... wrote %s" % file_content.filename)

    # Update and store max_date_time and known UUIDs
    # This is necessary to avoid downloading the same files over and over.
    if new_ral_files:
        max_date_time = ""
        uuids_starting_with_max_time = []

        for ral_file in new_ral_files:
            if ral_file.end_date_time > max_date_time:
                max_date_time = ral_file.end_date_time
                uuids_starting_with_max_time = []
            if ral_file.start_date_time == max_date_time:
                uuids_starting_with_max_time.append(ral_file.uuid)

        with open(config.target_dir + "/" + "last_call.json", mode="w") as f:
            json.dump({
                "uuids_starting_with_max_time": uuids_starting_with_max_time,
                "max_date_time": max_date_time
            }, f) 

 

Settings:

#
# Example configuration file for Read Access Log Download
#
# Edit it matching your configuration and save it as "settings.ini"
#

# Specify details on the ByDesign tenant
[BYD_TENANT]
hostname: my123456.sapbydesign.com
user: USER_NAME
password: PASSWORD
# if required, specify an http proxy. Comment out to use the environment variable HTTPS_PROXY.
# proxy: http://proxy.wdf.sap.corp:8080

# Specify the directory the log files should be written to
[TARGET]
dirname: ./log_entries 
To report this post you need to login first.

Be the first to leave a comment

You must be Logged on to comment or reply to a post.

Leave a Reply