Wayne 9:22 pm on February 12, 2020

GIT commands

GIT Normal commands ======================================================

git clone https://username:password@yourrepo.bitbucket.com

git status
git pull
git status
git add –all
git commit -m “comments”
git push -u origin master

Branches ======================================================
https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging
https://www.atlassian.com/git/tutorials/syncing/git-push
https://stackoverflow.com/questions/16408300/what-are-the-differences-between-local-branch-local-tracking-branch-remote-bra

— to view what is your remote
git remote -v
git branch -r — view remote branches
git branch -vv — view local branches

— switch to work on develop branch
git checkout -b develop — if you want to work on a new branch this should work for you:
git pull origin develop
git status
git add –all
git commit -m “comments”
git push -u origin develop

reset =========================================
git reset — give up all local changes

git reset –hard origin/develop — give up all local changes hard to a branch

Not track a file =========================================
Try using this command:

git update-index –assume-unchanged FILENAME_TO_IGNORE

To reverse it (if you ever want to commit changes to it), use:

git update-index –no-assume-unchanged

Here’s how to list ‘assume unchanged’ files under current directory:

git ls-files -v | grep -E “^[a-z]”

As the -v option will use lowercase letters for ‘assume unchanged’ files.

Conflict resolve ========================================================

if conflict, then edit conflicted file content, run git add on each file to mark it as resolved.

Staging the file marks it as resolved in Git.

Wayne 7:17 pm on July 26, 2019

Google Nature Language API

Here is a notes about the google nature language API summary I wrote three years ago. some feature may change a bit, refer to here for latest:

https://cloud.google.com/natural-language/docs/
Refer to here for basic concepts :
https://cloud.google.com/natural-language/docs/basics

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Features:

The Google Cloud Natural Language API provides natural language understanding technologies to developers, including:

sentiment analysis, – English
entity recognition, – English, Spanish, and Japanese – in fact is : nouns analysis
and syntax analysis – English, Spanish, and Japanese

API has three calls to do each, or can do them in one call, analyzeEntities, analyzeSentiment, annotateText.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
sentiment analysis

return the value to denote the text emotion negative/positive related extent only now.
has the polarity and magnitude value as the result.

entity recognition

find out the “entity” in text – prominent named “things” such as famous individuals, landmarks, etc.
return with entities and there URL/wiki, etc.

syntactic analysis

two things returns from syntax:
1. return the sentences/subsentences of input text.
2. return the the tokens (words) and there meta in grammar syntax dependency tree.

Test Steps of commands ++++++++++++++++++++++++++++++++++

gcloud auth activate-service-account --key-file=/yourprojecytkeyfile.json

gcloud auth print-access-token

print-access-token, This will give you a token for following commands. I create three json files to test each feature:
entity-request.json
syntactic-request.json
3in1-request.json.
so I can use these commands to try three API now:

curl -s -k -H "Content-Type: application/json" \
-H "Authorization: Bearer ya29.CjBWA2oWnup6dVvAlv6NTJyLsDtfqdCx70tX6_J0H7KFngd1ual2Osd8gCpcc" \
https://language.googleapis.com/v1beta1/documents:analyzeEntities \
-d @entity-request.json

curl -s -k -H "Content-Type: application/json" \
-H "Authorization: Bearer ya29.CjBWA2op6dVv_T7nAlv6NTJyLsDtfqdCx70tX6_J0H7KFngd1ual2Osd8gCpcc" \
https://language.googleapis.com/v1beta1/documents:annotateText \
-d @syntactic-request.json

curl -s -k -H "Content-Type: application/json" \
-H "Authorization: Bearer ya29.CjBWA2oWnup6_T7nAlv6NTJyLsDtfqdCx70tX6_J0H7KFngd1ual2Osd8gCpcc" \
https://language.googleapis.com/v1beta1/documents:annotateText \
-d @3in1-request.json

About how to create these json file as input, please refer to google SDK doc.

Wayne 7:03 pm on July 26, 2019

Google Cloud Basic

I have been using some google clouds API features , like ASR, TTS with wavnet, translate, calendar, for some years. My experience basically is that they have a clear product definition and clear API. But they also have some products not so many users and have to fade out from market. Comparing with AWS, google cloud is more technical oriented and AWS is good at facing the commercial company.

Here is a summary for some basic things to use google cloud:

First go to https://console.cloud.google.com/ to create one account and make some basic set up.
At the first, you can get some free volume , for some time for most API.
Create the project in the account and download the project private key as json file. later you will use this key.
Then you can start download the google cloud SDK, and this SDK may give you a lot commands of gcloud to do a lot works.
Many google API features can be tested through the gcloud in the command line. Normally you need to get a token key through the Oauth2 and use this key to run command API. Token will expire after some time.
So from this command line you can do immediate test for API.
If you use API client package, like java, python, js, you need to set env to refer to your project key for client package to run, like :
export GOOGLE_APPLICATION_CREDENTIALS=”/opt/googleSDK/myProject-ffe45577ca.json” , like set to env for java .
So client can run OK. Please refer to each API doc for how to use it.
If your free trial period is end, then you just need to upgrade your account to GCP billing account with proper billing information. then it will starting charging. After you active your upgrade, you do no need to update your project key json, cloud side will auto know your project is validated.
And at the manage console, you can see the API calling actives, calls per day, per minutes, etc. The manage console, looks like very complex, but in fact not too many things to manage over there, easy to use.

Wayne 9:51 pm on July 10, 2019

Docker common commands

https://docker-curriculum.com/

https://docs.docker.com/get-started/

Install

apt-get update
apt-get install docker.io sudo usermod -a -G docker $USER 
systemctl start docker
systemctl enable docker

then make a logoff , use groups to check you are in docker group or not:

groups

build image:

create a Dockerfile file and then run

docker build -t your_image .

list images and containers

docker images docker images-l

docker container ls --all

run a container and check the port:

docker run -d -p 8800:80 --name your_container your_image
docker port your_container

detached run the image as a container and map the 8800 port of the container to host 80 port

Then we can check that the new container with name based on ‘nginx_image’ is running, List the running containers:

docker ps

Or check all containers

docker ps -a
docker stop your_container
docker rm your_container
docker rmi your_image

Pull an image from a registry

docker pull alpine:3.4

Retag a local image with a new image name and tag

docker tag alpine:3.4 myrepo/myalpine:3.4

docker login my.registry.com:8000

Push an image to a registry

docker push myrepo/myalpine:3.4

Stop a running container through SIGTERM

docker stop my_container

Stop a running container through SIGKILL

docker kill my_container

Create an overlay network and specify a subnet

docker network create --subnet 10.1.0.0/24 --gateway 10.1.0.1 -d overlay mynet

List the networks

docker network ls

Delete all running and stopped containers

docker container prune Or this command in old versions: docker rm -f $(docker ps -aq)

 docker rm 305297d7a235 ff0a5c3750b9

Create a new bash process inside the container and connect it to the terminal

docker exec -it my_container bash

Print the last 100 lines of a container’s logs

docker logs --tail 100 my_container

docker run – Runs a command in a new container.
```
docker run hello-world
docker run -it busybox sh
```
docker start – Starts one or more stopped containers
docker stop – Stops one or more running containers
docker build – Builds an image form a Docker file
docker pull – Pulls an image or a repository from a registry

docker pull busybox

docker pull ubuntu:16.04

docker push – Pushes an image or a repository to a registry
docker export – Exports a container’s filesystem as a tar archive
docker exec – Runs a command in a run-time container
docker search – Searches the Docker Hub for images
```
docker search elasticsearch
docker search mysql
```
docker attach – Attaches to a running container
docker commit – Creates a new image from a container’s changes
Watch the log

docker container logs my_container

Wayne 9:33 pm on February 28, 2019

Spring boot build version and time access

Spring boot can create the build version info for you automatically and with a default bean to access these information. I found this link is very helpful on this and i jus tlist main points at here.

https://www.vojtechruzicka.com/spring-boot-version/

build-info goal create the version info

<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>build-info</goal>
</goals>
</execution>
</executions>
</plugin>

2. In java code, just use BuildProperties to get the version info. You cna then display on UI and logs.

@Autowired
BuildProperties buildProperties;

3. you can even add more properties in POM to let it saved in build-info, you can refer to the link details for this.

Wayne 12:46 am on February 8, 2019

How to let swagger control group display

In Swagger, you normally divide the API into groups and each group will have a Docket.

But many times, you do not want someone or some environment to see all groups . you want to control each group showing or not. This looks like a difficult task after some search. But in fact swagger already has this feature in Docket.

@Bean
public Docket myDocket() {
return new Docket(DocumentationType.SWAGGER_2).groupName("MYAPI").apiInfo(...).select().paths(....)
.build().useDefaultResponseMessages(false).forCodeGeneration(true).enable(true);
}

So enable(boolean) is part to control enable method for the group. You can use your configuration or role logic to control this flag value to control the group here.

Wayne 12:39 am on February 8, 2019

How to let your spring boot pick up external property file

You may already see that ./config/application.properties can override the default spring boot application.properties settings. But when you deploy your app to tomcat and with many other web app together, then you need to have unique properties files foe each app and also make external property file be picked up by web app.

So let us do something like this:

Change in your java main code to let your app use a unique properties file instead of application.properties.

@SpringBootApplication
public class MyApplication {

    public static void main(String[] args) {
        System.setProperty("spring.config.name", "myapp");
        SpringApplication.run(MyApplication.class, args);
    }
}

For war version of tomcat:

public class ServletInitializer extends SpringBootServletInitializer {

    @Override
    protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
        return application.sources(MyApplication.class).properties("spring.config.name: myapp");
    }

}

Then name your properties file like: myapp.properties, instead of the default name.

2. Now let us also enable the external property file to override the default myapp.properties.

in Spring boot, it is just ./config/myapp.properties

For tomcat version, we need to add env variable to do this. Set a spring_config_location environment variable pointing to the folder that contains your myapp.properties file. In the case of Tomcat you can do this by adding the following line to your <TOMCAT_HOME>/bin/setenv.sh file (create the file if missing):

export spring_config_location=<TOMCAT_HOME>/conf/

Place the myapp.properties properties file in that folder <TOMCAT_HOME>/conf/.

So every app will looking into the conf folder to look for their unique properties file to load.

Refer to:

https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html

Wayne 9:53 pm on February 7, 2019

Java native memory tools and usage related

https://dzone.com/articles/java-memory-and-cpu-monitoring-tools-and-technique

https://www.ibm.com/developerworks/library/j-nativememory-aix/

https://stackoverflow.com/questions/12797560/command-line-tool-to-find-java-heap-size-and-memory-used-linux

https://dzone.com/articles/java-memory-and-cpu-monitoring-tools-and-technique

https://stackoverflow.com/questions/2756798/java-native-memory-usage




java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'

Wayne 1:26 am on February 7, 2019

Issue: An irrecoverable stack overflow has occurred.

Issue:

An irrecoverable stack overflow has occurred. Please check if any of your loaded .so files has enabled executable stack (see man page execstack(8))

Reason:

This issue happens when I call a c++ So lib from java by JNA. This mostly caused by input a big chunk of data into the function interface of the C++ function. There is a limit of the Thread Stack Size defined for java’s thread. Normally is 512K-1024K.

And by change the Xss size you could change the stack size. but you should be careful on this when you have many threads need to open.

Refer http://xmlandmore.blogspot.com/2014/09/jdk-8-thread-stack-size-tuning.html