Google Nature Language API

Here is a notes about the google nature language API summary I wrote  three years ago. some feature may change a bit, refer to here for latest:

https://cloud.google.com/natural-language/docs/
Refer to here for basic concepts :
https://cloud.google.com/natural-language/docs/basics

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Features:

The Google Cloud Natural Language API provides natural language understanding technologies to developers, including:

• sentiment analysis, – English
• entity recognition, – English, Spanish, and Japanese – in fact is : nouns analysis
• and syntax analysis – English, Spanish, and Japanese

API has three calls to do each, or can do them in one call, analyzeEntities, analyzeSentiment, annotateText.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
sentiment analysis

return the value to denote the text emotion negative/positive related extent only now.
has the polarity and magnitude value as the result.

entity recognition

find out the “entity” in text – prominent named “things” such as famous individuals, landmarks, etc.
return with entities and there URL/wiki, etc.

syntactic analysis

two things returns from syntax:
1. return the sentences/subsentences of input text.
2. return the the tokens (words) and there meta in grammar syntax dependency tree.

 

Test Steps of commands ++++++++++++++++++++++++++++++++++

gcloud auth activate-service-account --key-file=/yourprojecytkeyfile.json

gcloud auth print-access-token

print-access-token, This will give you a token for following commands. I create three json files to test each feature:
entity-request.json
syntactic-request.json
3in1-request.json.
so I can use these commands to try three API now:

curl -s -k -H "Content-Type: application/json" \
-H "Authorization: Bearer ya29.CjBWA2oWnup6dVvAlv6NTJyLsDtfqdCx70tX6_J0H7KFngd1ual2Osd8gCpcc" \
https://language.googleapis.com/v1beta1/documents:analyzeEntities \
-d @entity-request.json

curl -s -k -H "Content-Type: application/json" \
-H "Authorization: Bearer ya29.CjBWA2op6dVv_T7nAlv6NTJyLsDtfqdCx70tX6_J0H7KFngd1ual2Osd8gCpcc" \
https://language.googleapis.com/v1beta1/documents:annotateText \
-d @syntactic-request.json

curl -s -k -H "Content-Type: application/json" \
-H "Authorization: Bearer ya29.CjBWA2oWnup6_T7nAlv6NTJyLsDtfqdCx70tX6_J0H7KFngd1ual2Osd8gCpcc" \
https://language.googleapis.com/v1beta1/documents:annotateText \
-d @3in1-request.json

About how to create these json file as input, please refer to google SDK doc.

 

 

 

Advertisements

Google Cloud Basic

I have been using some google clouds API features , like ASR, TTS with wavnet, translate, calendar, for some years. My experience basically is that they have a clear product definition and clear API. But they also have some products not so many users and have to fade out from market. Comparing with AWS, google cloud is more technical oriented and AWS is good at facing the commercial company.

Here is a summary for some basic things to use google cloud:

1. First go to https://console.cloud.google.com/ to create one account and make some basic set up.
2. At the first, you can get some free volume , for some time for most API.
3. Create the project in the account and download the project private key as json file. later you will use this key.
4. Then you can start download the google cloud SDK, and this SDK may give you a lot commands of gcloud to do a lot works.
5. Many google API features can be tested through the gcloud in the command line. Normally you need to get a token key through the Oauth2 and use this key to run command API.  Token will expire after some time.
   So from this command line you can do immediate test for API.
6. If you use API client package, like java, python, js, you need to set env to refer to your project key for client package to run, like :
   export GOOGLE_APPLICATION_CREDENTIALS=”/opt/googleSDK/myProject-ffe45577ca.json” , like set to env for java .
   So client can run OK. Please refer to each API doc for how to use it.
7. If your free trial period is end, then you just need to upgrade your account to GCP billing account with proper billing information. then it will starting charging. After you active your upgrade, you do no need to update your project key json, cloud side will auto know your project is validated.
8. And at the manage console, you can see the API calling actives, calls per day, per minutes, etc. The manage console, looks like very complex, but in fact not too many things to manage over there, easy to use.

 

 

 

 

Docker common commands

https://docker-curriculum.com/

https://docs.docker.com/get-started/

 

• Install

apt-get update
apt-get install docker.io sudo usermod -a -G docker $USER 
systemctl start docker
systemctl enable docker

then make a logoff , use groups to check you are in docker group or not:

groups

 

• build image:

create a Dockerfile file and then run

docker build -t your_image .

list images and containers

docker images docker images-l

docker container ls --all

run a container and check the port:

docker run -d -p 8800:80 --name your_container your_image
docker port your_container

detached run the image as a container and map the 8800 port of the container to host 80 port

Then we can check that the new container with name  based on ‘nginx_image’ is running, List the running containers:

docker ps

Or check all containers

docker ps -a
docker stop your_container
docker rm your_container
docker rmi your_image

Pull an image from a registry

docker pull alpine:3.4

Retag a local image with a new image name and tag

docker tag alpine:3.4 myrepo/myalpine:3.4

Log in to a registry (the Docker Hub by default)

docker login my.registry.com:8000

Push an image to a registry

docker push myrepo/myalpine:3.4

Stop a running container through SIGTERM

docker stop my_container

Stop a running container through SIGKILL

docker kill my_container

Create an overlay network and specify a subnet

docker network create --subnet 10.1.0.0/24 --gateway 10.1.0.1 -d overlay mynet

List the networks

docker network ls

Delete all running and stopped containers

docker container prune Or this command in old versions: docker rm -f $(docker ps -aq)

 docker rm 305297d7a235 ff0a5c3750b9

Create a new bash process inside the container and connect it to the terminal

docker exec -it my_container bash

Print the last 100 lines of a container’s logs

docker logs --tail 100 my_container

• docker run – Runs a command in a new container.

  docker run hello-world
  docker run -it busybox sh

   

• docker start – Starts one or more stopped containers
• docker stop – Stops one or more running containers
• docker build – Builds an image form a Docker file
• docker pull – Pulls an image or a repository from a registry

docker pull busybox

docker pull ubuntu:16.04

• docker push – Pushes an image or a repository to a registry
• docker export – Exports a container’s filesystem as a tar archive
• docker exec – Runs a command in a run-time container
• docker search – Searches the Docker Hub for images

  docker search elasticsearch
  docker search mysql

  docker attach – Attaches to a running container

• docker commit – Creates a new image from a container’s changes
• Watch the log

docker container logs my_container

Spring boot build version and time access

Spring boot can create the build version info for you automatically and with a default bean to access these information. I found this link is very helpful on this  and i jus tlist main points at here.

https://www.vojtechruzicka.com/spring-boot-version/

1. build-info goal create the version info

<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>build-info</goal>
</goals>
</execution>
</executions>
</plugin>

 

2. In java code, just use BuildProperties to get the version info. You cna then display on UI and logs.

@Autowired
BuildProperties buildProperties;

 

3. you can even add more properties in POM to let it saved in build-info, you can refer to the link details for this.

How to let swagger control group display

In Swagger, you normally divide the API into groups and each group will have a Docket.

But many times, you do not want someone or some environment to see all groups . you want to control each group showing or not. This looks like a difficult task after some search. But in fact swagger already has this feature in Docket.

@Bean
public Docket myDocket() {
return new Docket(DocumentationType.SWAGGER_2).groupName("MYAPI").apiInfo(...).select().paths(....)
.build().useDefaultResponseMessages(false).forCodeGeneration(true).enable(true);
}

So enable(boolean) is part to control enable method for the group. You can use your configuration or role logic to control this flag value to control the group here.

 

How to let your spring boot pick up external property file

You may already see that ./config/application.properties can override the default spring boot application.properties settings. But when you deploy your app to tomcat and with many other web app together, then you need to have unique properties files foe each app and also make external property file be picked up by web app.

So let us do something like this:

1. Change in your java main code to let your app use a unique properties file instead of application.properties.

@SpringBootApplication
public class MyApplication {

    public static void main(String[] args) {
        System.setProperty("spring.config.name", "myapp");
        SpringApplication.run(MyApplication.class, args);
    }
}

For war version of tomcat:

public class ServletInitializer extends SpringBootServletInitializer {

    @Override
    protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
        return application.sources(MyApplication.class).properties("spring.config.name: myapp");
    }

}

Then name your properties file like: myapp.properties, instead of the default name.

2. Now let us also enable the external property file to override the default myapp.properties.

in Spring boot, it is just ./config/myapp.properties

For tomcat version, we need to add env variable to do this. Set a spring_config_location environment variable pointing to the folder that contains your myapp.properties file. In the case of Tomcat you can do this by adding the following line to your <TOMCAT_HOME>/bin/setenv.sh file (create the file if missing):

export spring_config_location=<TOMCAT_HOME>/conf/

Place the myapp.properties properties file in that folder <TOMCAT_HOME>/conf/.

So every app will looking into the conf folder to look for their unique properties file to load.

 

Refer to:

https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html

 

 

Java native memory tools and usage related

 

https://dzone.com/articles/java-memory-and-cpu-monitoring-tools-and-technique

 

https://www.ibm.com/developerworks/library/j-nativememory-aix/

https://stackoverflow.com/questions/12797560/command-line-tool-to-find-java-heap-size-and-memory-used-linux

https://dzone.com/articles/java-memory-and-cpu-monitoring-tools-and-technique

 

 

https://stackoverflow.com/questions/2756798/java-native-memory-usage

 

 

java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'