macos system launch config

MacOS Config options

MacOS boot service is mainly configured via three ways:
1.  the system preferences -> Users & Groups -> Login items
2. /System/Library/StartupItems and /Library/StartupItems/
3. launchd system initialization process configuration.

The first two are relatively simple, we will focus mainly on the third more complex launchd configuration optimization.
launchd is a key process for initializing the system environment under Mac OS. Similar to Linux under init, rc.

MacOS Boot Flow

Let’s take a look at the MacOS startup flow:
1, mac firmware activation, initialize hardware, load BootX boot.
2, BootX load kernel and kernel extension (kext).
3, the kernel starts the launchd process.
4, launchd Start the service daemon by following the plist configuration in the /System/Library/LaunchAgents,  System/Library/LaunchDaemons, ?/Library/LaunchDaemons, ?Library/LaunchAgents, ~/Library/LaunchAgents.

The five launch DIRs

After going thru the Mac OS startup flow, we can see that the plist files in the below five directories  ?/System/Library/LaunchAgents, ?/System/Library/LaunchDaemons, ?/Library/LaunchDaemons, ?Library/LaunchAgents?, ~/Library/LaunchAgents. are key to the system optimization.

some basic concepts

difference between/System/Library and /Library and ~/Library directory?

/System/Library directory is stored in Apple’s own software development.
The /Library directory is the third-party software that the system administrator holds.
~/Library/ is the user’s own third-party software.

What is the difference between LaunchDaemons and LaunchAgents?

LaunchDaemons is the service (daemon) that the user started before the login.
LaunchAgents is the service (daemon) that the user starts after landing.

SPEC

plist file format and the meaning of each field in the above mentioned five directories:

KEY DESCRIPTION REQUIRED
Label The name of the job yes
ProgramArguments Strings to pass to the program when it is executed yes
UserName The job will be run as the given user, who may not necessarily be the one who submitted it to launchd. no
inetdCompatibility Indicates that the daemon anticip to be run as if it were launched by? Inetd no
Program The path to your executable. This key can save the ProgramArguments key for flags and arguments. no
onDemand A? Boolean? Flag that defined if a job runs continuously or not no
RootDirectory The job will be? Chrooted? Into another directory no
ServiceIPC Any the daemon can speak IPC to launchd no
WatchPaths Allows launchd to start a job based on modifications at a file-system path no
QueueDirectories Similar to WatchPath, a queue will only watch an empty directory for new files no
StartInterval Used to schedule a job that runs on a repeating schedule. Specified as the number of seconds to wait between runs. no
StartCalendarInterval Job scheduling. The syntax is similar to cron. no
HardResourceLimits Controls restriction on the resources consumed by any job no
LowPriorityIO Tells the kernel that this task is of a low priority when doing file system I / O no
Sockets An array can be used to specify what socket the daemon will listen on for launch on demand no

Do not understand the above plist configuration? It is fine. Our optimization strategy is completely unload the service, so we do not care that much about plist in the configuration meaning.

Leverage launchctl

To start to optimize the disabled service, we need to use the Mac OS provides a tool instruction- launchctl
The launchctl directive sets a disable flag for the service. When launchd starts, it checks whether the service is disabled or not to determine whether the service needs to be enabled.

Method 1 to disable service

First find the disabled flag file /var/db/launchd.db/com.apple.launchd/overrides.plist to see if the service you want to disable has been disabled.
Some services have been disabled but are not listed in overrides.plist. At this point, you also need to check the service plist file Label field has been marked as Disable.

After confirming that the service is not disabled, we can disable the service by invoking the following command:
sudo launchctl unload plist-file-path

For example, I would like to disable spotlight :

sudo launchctl unload /System/Library/LaunchAgents/com.apple.Spotlight.plist

After disabling the service, restart Mac OS to take effect.

Method of banning service 2

a more effective and violent method (recommended)
Uninstall the service first
sudo launchctl unload /System/Library/LaunchAgents/com.apple.Spotlight.plist
And then the plist file mv to other directory backup. Reboot. Done.

I personally prefer this way to disable the service, so recommend it.

If you find that the service is disabled, the system or software is abnormal, you can restore the service by putting the file back to the original folder and run the following command:
sudo launchctl load plist-file-path

Note: Be very careful when disabling System-level service,. please get familiar what that system service does and do enough research before remove/disable it. Otherwise it may cause the system to fail to start. The safest thing to do is to stop it.

On the other hand for the user service, we can rest assured that we could disable, in case there are problems we just need to re-enable it.

Here is a list of my disabled services:
/Library/LaunchDaemons/com.google.keystone.daemon.plist? (Google Software Update)
/Library/LaunchAgents/com.google.keystone.root.agent? (Google Software Update)
~ / Library / LaunchAgents / com.google.keystone.agent.plist? (Google Software Update, users do not need to add the process of sudo)
~ / Library / LaunchAgents / com.apple.CSConfigDotMacCert-ken.wug \ @ me.com-SharedServices.Agent.plist (me.com’s shared services, I do not have)
/System/Library/LaunchDaemons/org.cups.cupsd.plist (printer)
/System/Library/LaunchDaemons/org.cups.cups-lpd.plist (printer)
/System/Library/LaunchDaemons/com.apple.blued.plist (bluetooth)
/System/Library/LaunchAgents/com.apple.AirPortBaseStationAgent.plist (apple wireless base station, i do not have this device)

I know the daemon (service) name, how to find the corresponding plist file?
Copy the process (service) name, and then to / System / Library / LaunchAgents,? / System / Library / LaunchDaemons,? / Library / LaunchDaemons,? Library / LaunchAgents, ~ / Library / LaunchAgents five directories, through the following command :

ll | grep process(service)-name such as
ll | grep blued
Found it in /System/Library/LaunchDaemons Next, follow the steps outlined above to disable the service

For McAfee

For React Native local development in iOS, i have to disable Mcafee daemon to free the 8081 port. For android, we can do the workaround by using 8088.

 cd /Library/LaunchDaemons
 sudo launchctl unload com.mcafee.agent.macmn.plist
Advertisements

polygon in svg and css

in svg we can create triangle like below

<svg height="210" width="500">
  <polygon points="200,10 250,195 160,210" style="fill:lime;stroke:purple;stroke-width:1" />
  Sorry, your browser does not support inline SVG.
</svg>

turns out in css, we have similar stuff called clip-path which take similar parameter and draw the shape.

clip-path: polygon(50% 0%, 0% 100%, 100% 100%);

For circle/ellipse, it has specific function, circle and ellipse.

We can do something like

clip-path: circle(50% at 50% 50%);
Or
clip-path: circle(50px at 50px 50px) means at(50px, 50px),clip a circle with radius 50px.

More example here.

Or a 中文文章

debug hover item in chrome devtools

Chrome devtools is our friend, always.

Today when I was developing an angular 4.x app with primeng library, i have to check the class set on the tooltip component. As we know the tooltip is hover event based, so if we hover on it to make it showup and then shift our focus to the dev tool element tab, the tooptip would disappear.

Chrome tool has a feature to activate the hover stuff(:hover) on specific element for CSS sake. It is quite handy but obviously it does not apply in this use case since this tooltip is js based.

Search around and finally find a solution: using F8 or CMD + \ which is pause the script execution.

Steps are quite straightforward:

Mouse over the tooltip, and press F8 while it is displayed.

Now you can now use the inspector to look at the CSS.

LDAP notes on Forgerock OpenDJ

Forgerock has a good explanation on their openDJ, LDAP, DS etc…

Below are some of my notes.

LDAP directory data is organized into entries, similar to the entries for words in the dictionary, or for subscriber names in the phone book.

dn: uid=bjensen,ou=People,dc=example,dc=com
uid: bjensen
cn: Babs Jensen
cn: Barbara Jensen
facsimileTelephoneNumber: +1 408 555 1992
gidNumber: 1000
givenName: Barbara
homeDirectory: /home/bjensen
l: San Francisco
mail: bjensen@example.com
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: posixAccount
objectClass: top
ou: People
ou: Product Development
roomNumber: 0209
sn: Jensen
telephoneNumber: +1 408 555 1862
uidNumber: 1076

The entry also has a unique identifier, shown at the top of the entry, dn:uid=bjensen,ou=People,dc=example,dc=com. DN is an acronym for distinguished name. No two entries in the directory have the same distinguished name. Yet, DNs are typically composed of case-insensitive attributes.

When you look up her entry in the directory, you specify one or more attributes and values to match. The directory server then returns entries with attribute values that match what you specified.

A directory server stores two kinds of attributes in a directory entry: user attributes and operational attributes. User attributes hold the information for users of the directory. All of the attributes shown in the entry at the outset of this section are user attributes. Operational attributes hold information used by the directory itself. Examples of operational attributes include entryUUID, modifyTimestamp, and subschemaSubentry. When an LDAP search operation finds an entry in the directory, the directory server returns all the visible user attributes unless the search request restricts the list of attributes by specifying those attributes explicitly. The directory server does not, however, return any operational attributes unless the search request specifically asks for them. Generally speaking, applications should change only user attributes, and leave updates of operational attributes to the server, relying on public directory server interfaces to change server behavior. An exception is access control instruction (aci) attributes, which are operational attributes used to control access to directory data.

 

You may be used to web service client server communication, where each time the web client has something to request of the web server, a connection is set up and then torn down. LDAP has a different model. In LDAP the client application connects to the server and authenticates, then requests any number of operations, perhaps processing results in between requests, and finally disconnects when done.

各种license关系

licenses

GPL,以GPL为基础的软件也要用GPL,或者跟GPL兼容。有一个种方式可以做到不用GPL,把该软件版权持有者的公司,收购了,这是后话。目前GPL的主要流行版本是GPLv2 和GPLv3, 至于区别,可以理解为GPLv3有专利报复条款。

Apache License 比较宽松一些,简单可以理解为,在该授权软件基础上的软件可以不开源。

CDDL 可以理解为GPL 和Apache的折中,在一个软件中用不同几个包,在一个包里边,就是该比较完整的模块必须用CDDL,其他的可以用别的,甚至,不开源。

EPL则是因为后来IBM将Eclipse IDE交由名为“Eclipse基金会 (Eclipse Foundation)”来管理,对CPL为小部分修改为成的授权条款。EPL可以理解为在EPL授权的软件基础上的工作,如果新开的软件是源软件独立,就可以用其他的license,否则,只能用EPL。举个例子,你对EPL授权的软件,修正的bug,添加的性能提升,都不算独立的部分。

licenses2.jpg

large file from hive to rdbms(oracle)

Recently we have a requirement of dumping a sizable file(4+G) to oracle from s3. The file itself is hive-compatiable. so instead of downloading the file and generate sql for it, we decided to transfer the content using hive jdbc and persist in via jpa/hiberante.

Hive

On the hive side, one important thing is to make sure batchsize is set in jdbc resultset.


hiveJdbcTemplate.query(sqlToExecute, rs -> {
            rs.setFetchSize(5000);
            while (rs.next()){
               ....do you handling
            }
        });

RDBMS

on the relational database side

  1. make sure index is turned off. otherwise it each insertion will trigger the b-tree index to be inserted.
  2. make sure leverage the hibernate batch-size
    hibernate.jdbc.batch_size. I set it to 50 since my table has over 200 columns.For example , if you save() 100 records and your hibernate.jdbc.batch_size is set to 50. During flushing, instead of issue the following SQL 100 times :

    insert into TableA (id , fields) values (1, 'val1');
    insert into TableA (id , fields) values (2, 'val2');
    insert into TableA (id , fields) values (3, 'val3');
    .........................
    insert into TableA (id , fields) values (100, 'val100');

    Hiberate will group them in batches of 50 , and only issue 2 SQL to the DB, like this:

    insert into TableA (id , fields) values (1, 'val1') , (2, 'val2') ,(3, 'val3') ,(4, 'val4') ,......,(50, 'val50')
    insert into TableA (id , fields) values (51, 'val51') , (52, 'val52') ,(53, 'val53') ,(54, 'val54'),...... ,(100, 'val100')  

    Please note that Hibernate would disable insert batching at the JDBC level transparently if the primary key of the inserting table isGenerationType.Identity.

  3. make sure flush()/clear() for certain size so that memory is not eaten up by the millions of objects built on the fly.
    flush will make sure query be executed and object saved(synced) to DB.
    clear will clear the persistence context so all managed entities are detached. entities that have not been flushed will not be persisted.

My main code is something like:


    public int doImport(int limit)
    {
        String sql = "SELECT * FROM erd.ERD_PRDCT_FIXED_INCM_MNCPL_HS_prc_txt";
        if (limit >= 0)
        {
            sql = sql + " LIMIT " + limit;
        }
        HiveBeanPropertyRowMapper<SrcErdFixedIncmMuniEntity> mapper = new HiveBeanPropertyRowMapper<>(SrcErdFixedIncmMuniEntity.class);
        int batchSize = 5000;
        int[] inc = {0};
        Instant start = Instant.now();
        List<SrcErdFixedIncmMuniEntity> listToPersist = new ArrayList<>(batchSize);
        hiveJdbcTemplate.query(sql, (rs) -> {
            rs.setFetchSize(batchSize);
            while (rs.next())
            {
                listToPersist.add(mapper.mapRow(rs, -1));
                inc[0]++;
                if (inc[0] % batchSize == 0)
                {
                    persistAndClear(inc, listToPersist);
                }
            }
            //left overs(last n items)
            if(!listToPersist.isEmpty())
            {
                persistAndClear(inc, listToPersist);
            }
            return null;
        });
        Instant end = Instant.now();
        System.out.println("Data Intake took: " + Duration.between(start, end));
        return inc[0];
    }

    private void persistAndClear(int[] inc, List<SrcErdFixedIncmMuniEntity> listToPersist)
    {
        listToPersist.stream().forEach(em::persist);
        em.flush();
        em.clear();
        LOGGER.info("Saved record milestone: " + inc[0]);
        listToPersist.clear();
    }

Result

not bad, ~3.5 Millions records get loaded in about an hour.

serialize enum fields with gson

By default, Gson just serialize the ‘name’ of the Enum which might not be enough since we might need also want to carry all the fields during the serialization. To achieve this we need to has our own gson adaptor and make use of reflection.

public class EnumAdapterFactory implements TypeAdapterFactory
{

    @Override
    public <T> TypeAdapter<T> create(final Gson gson, final TypeToken<T> type)
    {
        Class<? super T> rawType = type.getRawType();
        if (rawType.isEnum())
        {
            return new EnumTypeAdapter<T>();
        }
        return null;
    }

    public class EnumTypeAdapter<T> extends TypeAdapter<T>
    {
        @Override
        public void write(JsonWriter out, T value) throws IOException
        {
            if (value == null || !value.getClass().isEnum())
            {
                out.nullValue();
                return;
            }

            try
            {
                out.beginObject();
                out.name("value");
                out.value(value.toString());
                Arrays.stream(Introspector.getBeanInfo(value.getClass()).getPropertyDescriptors())
                      .filter(pd -> pd.getReadMethod() != null && !"class".equals(pd.getName()) && !"declaringClass".equals(pd.getName()))
                      .forEach(pd -> {
                          try
                          {
                              out.name(pd.getName());
                              out.value(String.valueOf(pd.getReadMethod().invoke(value)));
                          } catch (IllegalAccessException | InvocationTargetException | IOException e)
                          {
                              e.printStackTrace();
                          }
                      });
                out.endObject();
            } catch (IntrospectionException e)
            {
                e.printStackTrace();
            }
        }

        public T read(JsonReader in) throws IOException
        {
            // Properly deserialize the input (if you use deserialization)
            return null;
        }
    }
}

usage:
Enum class:

public enum ReportTypes
{
    SP(1), CA(2), ADF(3), ORF(4), CTO(5), CDS(6), TSP(7);

    private int reportTypeId;
    ReportTypes(int reportTypeId)
    {
        this.reportTypeId = reportTypeId;
    }

    public int getReportTypeId()
    {
        return reportTypeId;
    }
}

Test Code:

    @Test
    public void testReportTypesGsonSerialization()
    {
        GsonBuilder builder = new GsonBuilder();
        builder.registerTypeAdapterFactory(new EnumAdapterFactory());
        Gson gson = builder.create();
        System.out.println(gson.toJson(ReportTypes.values()));
    }

Result:

[
  {
    "value": "SP",
    "reportTypeId": "1"
  },
  {
    "value": "CA",
    "reportTypeId": "2"
  },
  {
    "value": "ADF",
    "reportTypeId": "3"
  },
  {
    "value": "ORF",
    "reportTypeId": "4"
  },
  {
    "value": "CTO",
    "reportTypeId": "5"
  },
  {
    "value": "CDS",
    "reportTypeId": "6"
  },
  {
    "value": "TSP",
    "reportTypeId": "7"
  }
]