Skywalking-11: skywalking query protocol -- case study

Switch 2021-10-14 05:46:52

In the query Metrics  Information case analysis Skywalking  Query protocol

Basic overview

Skywalking  The query protocol is based on by default GraphQL , If necessary, you can also customize the extension , Provide an implementation of org.apache.skywalking.oap.server.core.query.QueryModule  The query module of .

Intercept Skywalking UI  Sent request

  • Request path
POST http://127.0.0.1:8080/graphql
  • Request body
{
"query": "query queryData($condition: MetricsCondition!, $duration: Duration!) {\n readMetricsValues: readMetricsValues(condition: $condition, duration: $duration) {\n label\n values {\n values {value}\n }\n }}",
"variables": {
"duration": {
"start": "2021-07-03 1320",
"end": "2021-07-03 1321",
"step": "MINUTE"
},
"condition": {
"name": "instance_jvm_thread_runnable_thread_count",
"entity": {
"scope": "ServiceInstance",
"serviceName": "business-zone::projectA",
"serviceInstanceName": "[email protected]",
"normal": true
}
}
}
}
  • Respond to
{
"data": {
"readMetricsValues": {
"values": {
"values": [
{
"value": 22
},
{
"value": 22
}
]
}
}
}
}

stay Skywalking  Find the corresponding... In the source code GraphQL  Definition

open oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol  Catalog , Use the template keyword in the request body readMetricsValues  Search for
stay oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol/metrics-v2.graphqls  Find the corresponding definition in

extend type Query {
# etc...
# Read time-series values in the duration of required metrics
readMetricsValues(condition: MetricsCondition!, duration: Duration!): MetricsValues!
# etc...
}

Input parameter definition

input MetricsCondition {
# Metrics name, which should be defined in OAL script
# Such as:
# Endpoint_avg = from(Endpoint.latency).avg()
# Then, `Endpoint_avg`
name: String!
# Follow entity definition description.
entity: Entity!
}
input Entity {
# 1. scope=All, no name is required.
# 2. scope=Service, ServiceInstance and Endpoint, set neccessary serviceName/serviceInstanceName/endpointName
# 3. Scope=ServiceRelation, ServiceInstanceRelation and EndpointRelation
# serviceName/serviceInstanceName/endpointName is/are the source(s)
# destServiceName/destServiceInstanceName/destEndpointName is/are destination(s)
# set necessary names of sources and destinations.
scope: Scope!
serviceName: String
# Normal service is the service having installed agent or metrics reported directly.
# Unnormal service is conjectural service, usually detected by the agent.
normal: Boolean
serviceInstanceName: String
endpointName: String
destServiceName: String
# Normal service is the service having installed agent or metrics reported directly.
# Unnormal service is conjectural service, usually detected by the agent.
destNormal: Boolean
destServiceInstanceName: String
destEndpointName: String
}
# The Duration defines the start and end time for each query operation.
# Fields: `start` and `end`
# represents the time span. And each of them matches the step.
# ref https://www.ietf.org/rfc/rfc3339.txt
# The time formats are
# `SECOND` step: yyyy-MM-dd HHmmss
# `MINUTE` step: yyyy-MM-dd HHmm
# `HOUR` step: yyyy-MM-dd HH
# `DAY` step: yyyy-MM-dd
# `MONTH` step: yyyy-MM
# Field: `step`
# represents the accurate time point.
# e.g.
# if step==HOUR , start=2017-11-08 09, end=2017-11-08 19
# then
# metrics from the following time points expected
# 2017-11-08 9:00 -> 2017-11-08 19:00
# there are 11 time points (hours) in the time span.
input Duration {
start: String!
end: String!
step: Step!
}
enum Step {
DAY
HOUR
MINUTE
SECOND
}

Return result definition

type MetricsValues {
# Could be null if no label assigned in the query condition
label: String
# Values of this label value.
values: IntValues
}
type IntValues {
values: [KVInt!]!
}
type KVInt {
id: ID!
# This is the value, the caller must understand the Unit.
# Such as:
# 1. If ask for cpm metric, the unit and result should be count.
# 2. If ask for response time (p99 or avg), the unit should be millisecond.
value: Long!
}

Use GraphQL IDEA  Plug in validation Skywalking UI  Request

Use “ GraphQL  stay Skywalking  Application in ” The way in the section , imitation “ Intercept Skywalking UI Sent request ” The request sent by the front end in the section

  • The request template
query queryData($condition: MetricsCondition!, $duration: Duration!) {
readMetricsValues: readMetricsValues(duration: $duration, condition: $condition) {
label values { values { id value }}
}
}
  • Request parameters
{
"duration": {
"start": "2021-07-03 1400",
"end": "2021-07-03 1401",
"step": "MINUTE"
},
"condition": {
"name": "instance_jvm_thread_runnable_thread_count",
"entity": {
"scope": "ServiceInstance",
"serviceName": "business-zone::projectA",
"serviceInstanceName": "[email protected]",
"normal": true
}
}
}
  • In response to the results
{
"data": {
"readMetricsValues": {
"values": {
"values": [
{
"id": "202107031400_YnVzaW5lc3Mtem9uZTo6cHJvamVjdEE=.1_ZThjZjM0YTFkNTRhNDA1OGE4Yzk4NTA1ODc3NzcwZTJAMTkyLjE2OC41MC4xMTM=",
"value": 22
},
{
"id": "202107031401_YnVzaW5lc3Mtem9uZTo6cHJvamVjdEE=.1_ZThjZjM0YTFkNTRhNDA1OGE4Yzk4NTA1ODc3NzcwZTJAMTkyLjE2OC41MC4xMTM=",
"value": 22
}
]
}
}
}
}

file

file

PS: If you don't use a template , There will be code prompts when writing query statements

query queryData {
readMetricsValues(
duration: {start: "2021-07-03 1400",end: "2021-07-03 1401", step: MINUTE},
condition: {
name: "instance_jvm_thread_runnable_thread_count",
entity: {
scope: ServiceInstance,
serviceName: "business-zone::projectA",
serviceInstanceName: "[email protected]",
normal: true
}
}
) {
label values{ values{ id value }}
}
}

How to integrate GraphQL Schema  Load the file into the program

Search for metrics-v2.graphqls , stay oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/GraphQLQueryProvider.java  Find the loading code

 // initialization GraphQL engine
@Override
public void prepare() throws ServiceNotProvidedException, ModuleStartException {
GraphQLSchema schema = SchemaParser.newParser()
// etc...
.file("query-protocol/metrics-v2.graphqls")
.resolvers(new MetricsQuery(getManager())) // MetricsQuery yes com.coxautodev.graphql.tools.GraphQLQueryResolver Interface implementation class
// etc...
.build()
.makeExecutableSchema();
this.graphQL = GraphQL.newGraphQL(schema).build();
}

stay org.apache.skywalking.oap.query.graphql.resolver.MetricsQuery  Class , find readMetricsValues  Method

 /**
* Read time-series values in the duration of required metrics
*/
public MetricsValues readMetricsValues(MetricsCondition condition, Duration duration) throws IOException {
if (MetricsType.UNKNOWN.equals(typeOfMetrics(condition.getName())) || !condition.getEntity().isValid()) {
final List<PointOfTime> pointOfTimes = duration.assembleDurationPoints();
MetricsValues values = new MetricsValues();
pointOfTimes.forEach(pointOfTime -> {
String id = pointOfTime.id(
condition.getEntity().isValid() ? condition.getEntity().buildId() : "ILLEGAL_ENTITY"
);
final KVInt kvInt = new KVInt();
kvInt.setId(id);
kvInt.setValue(0);
values.getValues().addKVInt(kvInt);
});
return values;
}
return getMetricsQueryService().readMetricsValues(condition, duration);
}
private MetricsQueryService getMetricsQueryService() {
if (metricsQueryService == null) {
this.metricsQueryService = moduleManager.find(CoreModule.NAME)
.provider()
.getService(MetricsQueryService.class);
}
return metricsQueryService;
}

org.apache.skywalking.oap.server.core.query.MetricsQueryService#readMetricsValues

 /**
* Read time-series values in the duration of required metrics
*/
public MetricsValues readMetricsValues(MetricsCondition condition, Duration duration) throws IOException {
return getMetricQueryDAO().readMetricsValues(
condition, ValueColumnMetadata.INSTANCE.getValueCName(condition.getName()), duration);
}
private IMetricsQueryDAO getMetricQueryDAO() {
if (metricQueryDAO == null) {
metricQueryDAO = moduleManager.find(StorageModule.NAME).provider().getService(IMetricsQueryDAO.class);
}
return metricQueryDAO;
}

see Extend storage file , IMetricsQueryDAO  Query data access objects for metrics

# Implement all DAOs
# Here is the list of all DAO interfaces in storage
IServiceInventoryCacheDAO
IServiceInstanceInventoryCacheDAO
IEndpointInventoryCacheDAO
INetworkAddressInventoryCacheDAO
IBatchDAO
StorageDAO
IRegisterLockDAO
ITopologyQueryDAO
IMetricsQueryDAO
ITraceQueryDAO
IMetadataQueryDAO
IAggregationQueryDAO
IAlarmQueryDAO
IHistoryDeleteDAO
IMetricsDAO
IRecordDAO
IRegisterDAO
ILogQueryDAO
ITopNRecordsQueryDAO
IBrowserLogQueryDAO

By class diagram , It can be seen that IMetricsQueryDAO  The implementation class has ES 、 ES7 、 InfluxDB 、 SQL  Four kinds of

file

How to integrate GraphQL  Engine registered to Jetty  service

 // register GraphQL Query the processor to Jetty service
@Override
public void start() throws ServiceNotProvidedException, ModuleStartException {
JettyHandlerRegister service = getManager().find(CoreModule.NAME)
.provider()
.getService(JettyHandlerRegister.class);
service.addHandler(new GraphQLQueryHandler(config.getPath(), graphQL));
}

Through analysis GraphQLQueryProvider  This kind , Discovery is QueryModule ( Query module ) Of Provider ( Provide ) class

thus , Also verified in “ Basic overview ” One section :

Skywalking The query protocol is based on by default GraphQL , If necessary, you can also customize the extension , Provide an implementation of org.apache.skywalking.oap.server.core.query.QueryModule The query module of .
 @Override
public String name() {
return "graphql";
}
@Override
public Class<? extends ModuleDefine> module() {
return QueryModule.class;
}
package org.apache.skywalking.oap.query.graphql;
import com.google.gson.Gson;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.reflect.TypeToken;
import graphql.ExecutionInput;
import graphql.ExecutionResult;
import graphql.GraphQL;
import graphql.GraphQLError;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.lang.reflect.Type;
import java.util.List;
import java.util.Map;
import javax.servlet.http.HttpServletRequest;
import lombok.RequiredArgsConstructor;
import org.apache.skywalking.oap.server.library.server.jetty.JettyJsonHandler;
import org.apache.skywalking.oap.server.library.util.CollectionUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@RequiredArgsConstructor
public class GraphQLQueryHandler extends JettyJsonHandler {
private static final Logger LOGGER = LoggerFactory.getLogger(GraphQLQueryHandler.class);
private static final String QUERY = "query";
private static final String VARIABLES = "variables";
private static final String DATA = "data";
private static final String ERRORS = "errors";
private static final String MESSAGE = "message";
private final Gson gson = new Gson();
private final Type mapOfStringObjectType = new TypeToken<Map<String, Object>>() {
}.getType();
private final String path;
private final GraphQL graphQL;
@Override
public String pathSpec() {
return path;
}
@Override
protected JsonElement doGet(HttpServletRequest req) {
throw new UnsupportedOperationException("GraphQL only supports POST method");
}
@Override
protected JsonElement doPost(HttpServletRequest req) throws IOException {
BufferedReader reader = new BufferedReader(new InputStreamReader(req.getInputStream()));
String line;
StringBuilder request = new StringBuilder();
while ((line = reader.readLine()) != null) {
request.append(line);
}
JsonObject requestJson = gson.fromJson(request.toString(), JsonObject.class);
return execute(requestJson.get(QUERY)
.getAsString(), gson.fromJson(requestJson.get(VARIABLES), mapOfStringObjectType));
}
private JsonObject execute(String request, Map<String, Object> variables) {
try {
ExecutionInput executionInput = ExecutionInput.newExecutionInput()
.query(request)
.variables(variables)
.build();
// Use GraphQL The engine gets the query results
ExecutionResult executionResult = graphQL.execute(executionInput);
LOGGER.debug("Execution result is {}", executionResult);
// Encapsulation return result
Object data = executionResult.getData();
List<GraphQLError> errors = executionResult.getErrors();
JsonObject jsonObject = new JsonObject();
if (data != null) {
jsonObject.add(DATA, gson.fromJson(gson.toJson(data), JsonObject.class));
}
if (CollectionUtils.isNotEmpty(errors)) {
JsonArray errorArray = new JsonArray();
errors.forEach(error -> {
JsonObject errorJson = new JsonObject();
errorJson.addProperty(MESSAGE, error.getMessage());
errorArray.add(errorJson);
});
jsonObject.add(ERRORS, errorArray);
}
return jsonObject;
} catch (final Throwable e) {
LOGGER.error(e.getMessage(), e);
JsonObject jsonObject = new JsonObject();
JsonArray errorArray = new JsonArray();
JsonObject errorJson = new JsonObject();
errorJson.addProperty(MESSAGE, e.getMessage());
errorArray.add(errorJson);
jsonObject.add(ERRORS, errorArray);
return jsonObject;
}
}
}

Webapp Gateway forwarding GraphQL  Request to OAP

v8.6.0  And before , All gateways are zuul , v8.7.0  And then replaced with Spring Cloud Gateway . Because this is not the focus of this article , No more details here

summary

Skywalking  By default, the query protocol of is highly universal GraphQL  Realization , The client can use the GraphQL  The protocol is very convenient to select the data you need .
Corresponding Skywalking  This pattern is relatively fixed 、 For infrequently changing query requirements , It's still quite suitable .

Reference documents

Please bring the original link to reprint ,thank
Similar articles

2021-10-14

2021-10-14

2021-10-14

2021-10-14

2021-10-14

2021-10-14

2021-10-14